Senior ML OPs Engineer

Role Overview

Senior MLOps Engineer (Applied AI Focus) on the Product Innovations team. You will act as the technical lead for Applied MLOps, bridging experimental research to production-ready AI systems. The work focuses on ground-truth generation, model evaluation, and the pre-/post-processing infrastructure behind large-scale vector embeddings and applied AI features.

Responsibilities

• Architect and implement annotation/measurement pipelines
  • Design human-in-the-loop and automated annotation workflows
  • Build systems for quality metrics, confidence scoring, and inter-annotator agreement (IAA)
  • Stand up annotation tools and processes for ground-truth generation at scale
• Drive cost-efficient model evaluation
  • Create ground-truth datasets, golden sets, and evaluation criteria per model
  • Benchmark model performance to guide data science and optimize compute and cost
  • Help deploy the right model for the right use case to support product margins
• Enforce Applied MLOps standards
  • Implement deterministic preprocessing, PII scrubbing, and model-as-a-judge evaluation loops
  • Establish production standards for safety, consistency, and reliability
• Collaborate on model & product strategy
  • Partner with Data Science on applied generative AI decisions
  • Influence model selection, architecture, and optimization approaches
• Build and integrate with infrastructure
  • Integrate measurement/evaluation loops into cloud ecosystems (AWS/GCP)
  • Deliver automation and observability across the model lifecycle

Requirements

• Experience building/managing annotation workflows (commercial or custom tools)
• Strong knowledge of quality metrics, ground-truth processes, and IAA
• Hands-on applied MLOps experience with monitoring, versioning, evaluation frameworks, and production ML infrastructure
• Strong Python skills for integrations, pipelines, and ML tooling
• Cloud ML experience with AWS or GCP (or equivalent)
• Ability to collaborate effectively with DevOps/SRE teams
• Collaborative technical leadership and cross-functional communication

Nice-to-Haves / Signals

• Emphasis on pragmatic execution and scalable solutions (not perfection)
• Understanding of trade-offs among model size, cost, latency, and accuracy

Work Model & Location

• Remote with hybrid collaboration available
• Located in Los Angeles County, CA (posting location shown)

Compensation

• $198,000 – $230,000 (annual)
• No sponsorship available