Tech Lead Databricks Data Engineer

Role Overview

Tech Lead, Data Engineer (Databricks) at Mitre Media. Reporting to the CTO, you will architect and maintain the data backbone powering features across the product suite—designing Databricks-based ETL pipelines, modeling investment data, and delivering low-latency, high-quality datasets for both user-facing features and internal AI/analytics workloads.

Responsibilities

• Design, implement, and optimize large-scale ETL workflows in Databricks using Apache Spark, Delta Lake, and dbt.
• Build algorithms to transform raw market data into actionable insights.
• Own data quality and lineage with tests, monitoring, and alerting for mission-critical pipelines.
• Evolve the cloud data platform on AWS and GCP for scale, performance, and cost efficiency.
• Mentor engineers and raise standards through code reviews, documentation, and data/DevOps best practices.
• Enable AI workflows by building feature stores or inference-ready tables for ML/LLM systems.

Required Qualifications

• Python expertise; working knowledge of Scala or Java.
• Hands-on experience with Databricks & Spark including cluster tuning, job orchestration, and Delta Tables.
• Strong SQL and dbt; modular data-model design.
• Experience with CI/CD for transformations.
• Production experience on AWS or GCP data services (e.g., S3/GCS, EMR/Dataproc, Glue/Dataflow).
• Solid understanding of EL(T) patterns, workflow scheduling, and incremental processing.
• Data modeling experience including dimensional modeling and schema-on-read designs.

Nice-to-Haves

• DAG orchestration tools such as Apache Airflow, Luigi, or Dagster.
• Visualization tooling experience (e.g., Looker, Tableau, or Stripe Vizier-style stacks).
• Financial markets domain knowledge (equities, ETFs, mutual funds).
• Background in machine learning engineering or statistical analysis.
• Familiarity with vector databases or embedding pipelines.

Soft Skills

• Entrepreneurial, AI/FinTech-driven mindset; proactive problem diagnosis and trade-off decisions.
• Clear communicator in distributed cross-functional teams.
• Mentorship mindset (uplifts peers via code reviews and knowledge sharing).
• Detail-oriented with strong discipline for data accuracy and documentation.

Compensation & Location

• $150k–$185k CAD/USD plus benefits and performance bonuses.
• Remote-first within ±3 hours of Eastern Time; optional collaboration hubs in Toronto and NYC.