Data Engineer

Role Overview

We're seeking a highly skilled Data Engineer to design, build, and maintain production-grade data pipelines that process and transform terabytes of data. You'll collaborate closely with data scientists and other software engineers to ensure our data infrastructure is scalable, reliable, and cost-effective.

Key Responsibilities

Pipeline Development & Deployment

• Architect, develop, and deploy batch and streaming pipelines using Airflow and containerized workflows for cyber-security use-cases
• Containerize data-processing jobs with Docker, orchestrate with Kubernetes, and manage releases with Helm charts

Distributed Computing

• Build high-throughput data transformations using Dask or Apache Spark
• Maintain training data clusters across hybrid on-prem and cloud environments
• Optimize training jobs for performance, resiliency, and cost

Monitoring & Reliability

• Implement observability (logging, metrics, alerting) to maintain pipeline health and SLA adherence
• Troubleshoot, debug, and resolve data-processing failures in production

Collaboration & Best Practices

• Work with cross-functional teams to define data contracts, schemas, and quality checks
• Enforce software engineering best practices: CI/CD, code reviews, automated testing, and documentation

Data Modeling & Storage

• Design and maintain data models and schemas for AI/ML continuous training use cases
• Load data into cloud storage and lakes, ensuring performance and accessibility

Requirements

• 3–5 years of professional experience designing and operating production data pipelines at scale
• Expertise with Docker, Kubernetes, and Helm
• Hands-on experience building DAG-based pipelines in Apache Airflow
• Strong proficiency in Python for data engineering tasks
• Practical experience with Dask or Apache Spark for large-scale data processing
• Familiarity with deploying and managing services in a cloud environment
• Hands-on with Google Cloud services (Pub/Sub, BigQuery, Cloud Storage, GKE)
• Knowledge of data governance, encryption, and role-based access control

Nice-to-Haves

• Experience writing data services in Go or Rust
• Exposure to deploying cross-cluster model-training workflows using Ray or similar frameworks
• Familiarity with Terraform for infrastructure as code
• Experience with other public cloud providers (AWS, Azure, etc.)