Quality Assurance Lead

Role Overview

Quality Assurance Lead (Principal QA Engineer — AI Systems & Platform) at Peach Pilot. This is a founding QA hire responsible for building and owning the QA function from scratch for an early engineering team.

Mission

Trust has to be earned: every release must work exactly as users expect, because enterprise AI pilots often fail due to user distrust rather than broken technology.

What You Will Own & Build

First 90 Days: Build the QA Foundation

• Establish the testing framework from zero:
  • unit, integration, end-to-end tests
  • LLM-specific evaluation pipelines
• Define quality standards, test coverage requirements, and documentation practices with the Lead Engineer
• Audit the platform and identify highest-risk surfaces before major deployments
• Define an onshore vs. offshore QA team structure and execute an initial hiring roadmap

Build and Lead the QA Team

• Recruit, hire, and onboard QA engineers as the team grows
• Set expectations, working standards, and a bar for technical excellence
• Mentor junior and mid-level QA engineers so they can independently own test domains
• Establish a company-wide quality culture (QA as everyone’s responsibility)
• Report directly to the Lead Engineer and participate in product planning so quality is built in

AI & Agent Testing

• Design evaluation frameworks for non-deterministic LLM outputs, including:
  • prompt regression testing
  • model drift detection
  • output quality scoring across Claude, GPT-4o, Grok, Gemini
• Build automated test suites for the agent orchestration layer, including:
  • governance agent audit trail integrity
  • human-override behavior
• Validate the Enterprise Knowledge Graph (Neo4j + vector search) for:
  • data accuracy
  • retrieval quality
  • failure modes under real enterprise data conditions

Platform & Integration Testing

• Own end-to-end testing of the file ingestion pipeline across:
  • Word, Excel, PowerPoint, PDF
• Cover encryption, formatting edge cases, and audit trail continuity

Requirements

• Ability to build QA strategy and test infrastructure from scratch (not a ticket-closing role)
• Deep understanding of testing approaches for non-deterministic systems (LLMs/agents)
• Experience designing and implementing automated evaluation pipelines and end-to-end quality checks
• Ability to lead a QA function and mentor engineers

Nice-to-Haves

• Familiarity with enterprise AI patterns: multi-model routing, governance/audit trails, and knowledge graph + vector retrieval
• Hands-on testing experience for document ingestion workflows and related edge cases