QA/Test Engineer (Train AI Models Part Time!)

Role Overview

Join a leading AI lab’s GenAI team as a QA/Test Engineer focused on agentic evaluation benchmarks for frontier models. You’ll ensure that complex multi-step tasks are unambiguous, correctly graded, and robust to shortcuts, acting as the quality backbone for benchmark reliability.

Responsibilities

• Design checks: Create test cases that verify tasks work as intended, including edge cases.
• Review tasks: Carefully review tasks and reference solutions to catch ambiguity and gaps before finalization.
• Debug: Use Python to troubleshoot tasks and quality checks when behavior is incorrect.
• Shape the process: Help build simple, repeatable quality checklists and provide actionable feedback to task authors.
• Protect results: Identify and prevent shortcutting and grading gaps so benchmark scores remain trustworthy.

Core Qualifications

• MSc/PhD in a STEM field or equivalent practical experience in research/engineering-heavy domains.
• 1+ years in test engineering, QA, or research/software engineering with strong quality ownership.
• Proven ability to design test cases and quality review processes, and debug complex systems end-to-end.
• Working proficiency in Python and Git; comfortable navigating unfamiliar codebases/environments.
• Exceptional attention to detail and strong written documentation habits.

Preferred

• Experience with AI training, model evaluation, or quality review of AI-generated work.
• A perfectionist mindset and ability to work independently through ambiguous, open-ended problems.

Work Setup

• Fully remote within the United States.
• Approximately 35 hours/week.