ML Engineer

Role Overview

Senior MLOps Engineer (Applied AI) for the Product Innovations team. You will lead applied MLOps initiatives by bridging experimental data science work with production-ready, scalable solutions.

Responsibilities

• Architect and implement annotation + measurement pipelines, including human-in-the-loop and auto-annotation workflows.
• Optimize ground truth generation and establish confidence metrics and Inter-Annotator Agreement (IAA) standards.
• Build and maintain golden datasets for benchmarking and evaluation.
• Define evaluation criteria and benchmarking processes to support model selection and deployment decisions.
• Implement applied MLOps practices such as:
  • Model-as-a-judge frameworks
  • Deterministic PII scrubbing
  • Compliance protocols
• Collaborate with data scientists to develop strategies for generative AI models and improve model performance.
• Integrate measurement and evaluation loops into cloud infrastructure (AWS, GCP) for automated model lifecycle management.
• Partner with engineering teams to ensure deployment, monitoring, and observability in production.
• Drive continuous improvements in cost efficiency, scalability, and robustness.

Requirements

• Deep experience setting up annotation workstreams using tools like Label Studio and Scale AI (or custom solutions).
• Strong applied MLOps expertise: monitoring, versioning, evaluation, and production deployment.
• Proficiency in Python for scripting, API integration, and automation.
• Understanding of optimization trade-offs across performance, latency, and cost.
• Cloud experience with AWS (SageMaker, S3) and GCP (Vertex AI).
• Knowledge of data pipeline architecture and infrastructure automation.
• Strong collaboration and communication; ability to translate technical concepts into practical solutions.
• Degree in Computer Science/Data Science/Engineering or related field (advanced degree preferred).

Nice-to-haves (implied)

• Experience with generative AI measurement/evaluation frameworks.
• Experience implementing compliance-focused PII handling and deterministic scrubbing.