Senior Software Engineer (Data Governance & Foundations)

Role Overview

Join Instacart’s Data Governance & Foundations team as a Senior Software Engineer. You’ll own and maintain core systems powering the company’s data ecosystem—defining multi-year architecture roadmaps, optimizing infrastructure spend, and mentoring engineers in a high-autonomy environment.

Key Missions

• Define and drive multi-year architecture roadmaps for large-scale data ingestion and processing infrastructure, balancing reliability, scalability, and cost.
• Own end-to-end platform initiatives across:
  • Kafka-based streaming
  • Postgres-based systems
  • Build vs. buy decisions and migration design
  • Production rollout and risk management
• Embed data governance and compliance controls into platform architecture and data lifecycle management, including:
  • SOX, CPRA, GDPR

Responsibilities

• Architect and maintain foundational data platform systems for evolving business and compliance needs.
• Partner with engineering leadership and cross-functional teams on platform strategy and execution.
• Provide clear technical communication (architecture docs, strategy memos, and proposals).
• Mentor engineers and collaborate to ensure the platform meets long-term requirements.

Requirements

• 5+ years of software engineering focused on data infrastructure and/or distributed systems at scale.
• Strong event-driven infrastructure experience:
  • Kafka for high-throughput ingestion
  • Flink (or equivalent) for stream processing at scale
• Proven ability to lead major platform transitions (migration design, phased rollout, risk management).
• Hands-on experience with distributed query/compute systems and production reliability:
  • Trino, Spark, ClickHouse (including performance tuning)
• Modern data lakehouse and open table formats:
  • Apache Iceberg, Delta Lake, Hudi (including compute/storage trade-offs)
• Ability to build infrastructure business cases (cost-benefit, TCO modeling, leadership recommendations).
• Strong ownership and ability to operate in ambiguity, driving multi-team initiatives to production.
• Data governance and compliance framework familiarity:
  • SOX, CPRA, GDPR
• FinOps and data platform cost optimization experience (large budgets, vendor negotiation).
• Deep SQL expertise and proficiency in Python or Scala.
• Education in CS/related fields (BS/MS/PhD) or equivalent practical experience.
• Experience with:
  • Apache Airflow (orchestration)
  • dbt (transformation pipelines) in large-scale production environments

Nice-to-haves

• Direct ownership of governance-compliant ingestion/processing pipelines and operational cost optimization across enterprise-scale systems.