Data Engineer

Role: Data Engineer

Build and maintain production-grade data pipelines that process and transform large volumes of data (terabytes), supporting cyber-security and AI/ML training use cases.

Responsibilities

• Pipeline development & deployment:
  • Architect, develop, and deploy batch and streaming pipelines using Apache Airflow and containerized workflows.
  • Containerize data-processing jobs with Docker, orchestrate with Kubernetes, and manage releases with Helm.
• Distributed computing:
  • Implement high-throughput data transformations using Dask or Apache Spark.
  • Maintain training data clusters across hybrid (on-prem + cloud) environments.
  • Optimize training jobs for performance, resiliency, and cost.
• Monitoring & reliability:
  • Add observability (logging, metrics, alerting) to maintain pipeline health and SLA adherence.
  • Troubleshoot and resolve production data-processing failures.
• Collaboration & best practices:
  • Define data contracts, schemas, and data quality checks with cross-functional teams.
  • Apply software engineering best practices: CI/CD, code reviews, automated testing, and documentation.
• Data modeling & storage:
  • Design and maintain data models/schemas for AI/ML continuous training.
  • Load data into cloud storage/data lakes with performance and accessibility in mind.

Requirements

• 3–5 years of experience designing and operating production data pipelines at scale.
• Strong Python skills for data engineering.
• Docker, Kubernetes, Helm expertise.
• Hands-on Apache Airflow experience with DAG-based pipelines.
• Experience with Dask or Apache Spark for large-scale processing.
• Cloud fundamentals and deploying/managing services in a cloud environment.
• Practical Google Cloud experience (e.g., Pub/Sub, BigQuery, Cloud Storage, GKE). Other public clouds acceptable.
• Infrastructure as Code familiarity with Terraform.
• Knowledge of data governance, encryption, and role-based access control (RBAC).
• Exposure to Ray (or similar) for cross-cluster model-training workflows.
• Experience writing data services in Go or Rust.

Additional Info

• Location: United States. Applicants must be authorized to work in the US.