Data Engineer

Role Overview

Design, build, and optimize the core data infrastructure and pipelines that power business intelligence and machine learning models. You will process high volumes of structured and unstructured data daily, collaborating with data scientists, software engineers, and business stakeholders to ensure data availability, reliability, and security.

Key Responsibilities

• Design, build, and maintain scalable batch and real-time data pipelines using Apache Spark, dbt, and Airflow
• Optimize data warehouse performance in Snowflake, BigQuery, or Redshift (e.g., clustering keys, partitioning, query optimization)
• Implement robust data quality checks, anomaly detection, and schema validation across ingestion and transformations
• Collaborate with backend teams to integrate event streams via Kafka, Kinesis, or Pub/Sub into a central lakehouse
• Use infrastructure-as-code to deploy pipelines with Terraform and maintain CI/CD using GitHub Actions
• Monitor pipeline health, latency, and cloud compute costs, applying optimizations to reduce infrastructure spend

Requirements

• 3–6 years of data engineering experience in a cloud-native production environment (AWS, GCP, or Azure)
• Strong SQL and Python skills and a solid grasp of software engineering best practices/patterns
• Hands-on experience with dbt for complex transformations and orchestration with Airflow or Prefect
• Proven experience with distributed compute (e.g., Apache Spark, Databricks, AWS EMR)
• Solid understanding of data modeling (e.g., Kimball dimensional modeling, Data Vault, wide-table strategies)

Bonus

• Experience with real-time streaming architectures (e.g., Kafka, Flink)
• Experience managing infrastructure with Terraform