DevOps Engineer (Full Time - Remote)

Role Overview

DevOps Engineer focused on authoring and validating Infrastructure-as-Code (IaC) tasks for AI agent evaluation. You will design realistic AWS/GCP scenarios, create ground-truth infrastructure solutions, and implement automated checks to verify whether an AI agent reaches the correct end state.

Responsibilities

• Author IaC tasks grounded in real-world AWS and GCP scenarios
• Build correct, idempotent IaC solutions for each task
• Create verifiable automated graders that validate the AI agent’s outcome
• Review and QA IaC tasks authored by other engineers (correctness, difficulty, robustness)
• Harden tasks against reward hacking
• Document task intent, assumptions, edge cases, and scoring rationale

Requirements

• 4+ years of DevOps / cloud engineering experience
• Deep, hands-on IaC expertise:
  • Terraform (required)
  • Pulumi, CloudFormation, or CDK (preferred/plus)
• Strong AWS and GCP knowledge
• Comfortable scripting in Python and/or Bash for automation and validation
• Strong focus on correctness and verification (ability to articulate “right and safe” infrastructure outcomes in code)

Preferred Qualifications

• Prior work in AI evaluation, benchmarking, or expert data annotation
• Experience with LocalStack