Staff Machine Learning Engineer

Role Overview

Join Toast as a Staff Machine Learning Engineer on the ML Platform team. You will be a technical leader responsible for the architecture and foundational infrastructure that power Toast’s machine learning capabilities.

Responsibilities

• Provide technical leadership for the ML Platform team and shape platform systems for ML capabilities
• Drive architectural direction across the ML platform
• Deliver foundational ML infrastructure and platform initiatives across large scope
• Lead design and delivery of major platform work streams, coordinating with multiple teams
• Elevate and mentor fellow engineers; drive cross-team alignment with engineering, product, and business stakeholders

Requirements

• 8+ years delivering complex backend/infrastructure systems at scale
• Strong understanding of data modeling, query languages, and online/offline data patterns for ML systems
• Mastery of a modern backend language: Python, Java, Kotlin, Go, or Scala
• Direct experience building or operating core ML infrastructure, such as:
  • Feature stores
  • Model serving
  • Experimentation platforms
  • Training orchestration
• Demonstrated technical leadership and ability to influence across teams
• Deep proficiency in distributed systems (consistency, latency, throughput, fault tolerance, observability)
• Hands-on experience with ML platform components such as Tecton, MLflow, SageMaker, or Databricks
• Experience building/operating experimentation / A-B testing platforms at scale

Nice-to-haves / Additional Skills

• Familiarity with real-time streaming systems for feature computation: Kafka, Flink, Spark Streaming
• Experience serving LLMs/large deep-learning models in production (GPU capacity planning, inference optimization)
• Experience supporting internal developer-facing platforms with a product mindset
• Comfort with Kubernetes and modern cloud-native infrastructure