Senior Software Engineer (Python) - Agent Evaluation - Freelance/Remote 100+ openings

Role overview

Senior Python Engineer for Agent Evaluation (freelance/remote). You will help build a dataset to evaluate AI coding agents, by creating realistic developer tasks and the evaluation tests that measure agent performance on real-world scenarios.

What you’ll do

• Create realistic developer environments: a virtual company with codebase, infrastructure, and context (tickets, docs, conversations).
• Design tasks from intermediate states: craft prompts and define what it means to be “solved,” ensuring tasks are realistically solvable by an AI agent.
• Write automated tests for agent solutions: accept valid approaches and reject incorrect ones; keep strictness balanced.
• Iterate with QA feedback: review agent outcomes, analyze failures, and refine tasks/tests to improve fairness and robustness.

What this is NOT

• Not data labeling
• Not prompt engineering
• Not writing code from scratch (the agent writes most of the code; you guide and evaluate)

Requirements

• 5+ years professional experience with Python
• Strong experience with FastAPI, pytest, and async/await
• Hands-on experience with Docker, PostgreSQL, and CI/CD pipelines
• Proven experience writing and maintaining automated tests (not only executing them)
• English proficiency: B2 or higher
• Availability 30+ hours/week

Nice to have

• Full-stack experience with React and TypeScript

Remote & location

• Open to candidates in North America, South America, Asia, and Europe
• Remote work; reliable internet connection required

How it works / onboarding

• Apply → qualification(s) → join project → complete tasks → get paid
• Includes app review, virtual intro session, technical assessment, identity verification, and a background check
• Participation in Discord for communication and updates