Senior AI Quality Engineer

Role Overview

Senior AI Quality Engineer focused on ensuring AI-powered agentic applications are reliable, observable, and safe to operate in mission-critical environments. You’ll design testing guardrails, validate orchestration logic, and build bespoke evaluation and test harnesses based on how these systems actually behave.

Responsibilities

• Hands-on delivery: Contribute across the AI lifecycle (requirements/design → development → testing → deployment), writing code, reviewing PRs, and delivering iterative AI quality capabilities in an agile environment.
• Scenario-driven validation: Create and maintain systems that run agentic workflows against realistic scenarios, edge cases, and failure conditions using custom harnesses and guardrail-style validations.
• Integration hardening: Define interface contracts, validation, and explicit failure behavior so workflows degrade safely when dependencies misbehave.
• Reusable quality components: Build shared libraries, reference harnesses, and validation utilities that teams can reuse to adopt AI quality practices faster.
• Quality by design: Influence architecture so correctness, safety, and expected behavior are enforced by design.
• Design-time risk reduction: Identify likely agentic failure patterns early and shape architecture and constraints before implementation.
• AI quality evaluation: Implement cost-conscious evaluation strategies covering data quality validation, model relevancy, hallucination detection, failure analysis, and ongoing measurement.
• Mission-aligned evaluation: Partner with engineering, infrastructure, security, and delivery stakeholders to embed quality signals across systems from individual agents to end-to-end workflows.

Requirements

• Security clearance eligibility: Must be eligible to obtain a security clearance; non-US citizens may not be eligible.
• Strong understanding of agentic system design and failure modes in AI-driven workflows.
• Comfort building bespoke testing frameworks, simulators, and evaluation pipelines.

Nice-to-Haves

• Experience defining quality measurement and enforcement mechanisms across individual agents to end-to-end workflows.
• Experience collaborating across engineering, infrastructure, and security to embed reliability, security, and evaluation signals.