AI Tester - Remote (US) - Only W2

Role Overview

You will be a hands-on QA Engineer for a GenAI-based, multi-agent chatbot platform. The work focuses on testing LLM-driven behavior, RAG outputs, security, and cloud-native integrations on AWS.

Responsibilities

• Develop and execute test plans for chatbot logic, GenAI responses, and agent interactions.
• Validate RAG functionality including:
  • data retrieval accuracy
  • response relevance
  • hallucination detection
• Test API integrations between the chatbot, AWS services, and enterprise systems.
• Implement automation for regression, performance, and cost-efficiency testing.
• Verify LLM prompt security (e.g., guardrails and injection testing).
• Support analytics validation for:
  • user engagement
  • response accuracy
  • model performance metrics.

Requirements

• Hands-on QA/testing experience for LLM/GenAI features and multi-agent systems.
• Ability to validate RAG behavior and LLM security (prompt/guardrail/injection testing).
• Experience testing API integrations and ensuring reliability and performance in AWS environments.

Nice-to-haves

• Experience with chatbot agent frameworks such as LangGraph or AWS Strands Agents.
• Automation experience focused on regression, performance, and cost testing.