Senior Software Engineer (Python) - Agent Evaluation - Freelance/Remote 100+ openings

Role overview

As a Senior Software Engineer (Python), you will help build a dataset to evaluate AI coding agents by creating realistic developer tasks and the evaluation criteria around them. You’ll design environments, define what “solved” means, and implement tests that judge agent responses.

What you’ll do

• Build realistic developer environments: simulate a “virtual company” with a codebase, infrastructure, and context (tickets, docs, conversations).
• Design tasks from intermediate states: craft prompts and define solution criteria, ensuring tasks are solvable by an AI agent.
• Write automated tests: accept all valid solutions and reliably reject incorrect ones (balanced strictness).
• Iterate with QA feedback: review failures, analyze where agents break, and refine tasks/tests to make evaluation robust and fair.

What this is not

• Not data labeling
• Not prompt engineering
• Not writing code from scratch (the agent writes most of the code; you guide and evaluate)

Requirements

• 5+ years professional experience with Python
• Strong experience with FastAPI, pytest, and async/await
• Hands-on with Docker, PostgreSQL, and CI/CD pipelines
• Proven experience maintaining automated tests (beyond just running them)
• React + TypeScript experience is a plus
• English proficiency: B2 or higher
• Availability 30+ hours/week

Process & logistics

• Apply → pass qualification(s) → join a project → complete tasks → get paid
• Virtual project introduction, technical assessment, identity verification, onboarding training
• Background check required prior to onboarding
• Participate in Discord for project communication
• Reliable internet connection for remote work

Location / eligibility

Open to candidates in North America, South America, Europe, and Asia; CV must be submitted in English.