xelys jobs xelys jobs

Staff AI Engineer

Grafana Labs

full-remoteleadpermanentbackenddata Full remote 194 days ago via WTTJ

See how well this job matches your profile

Sign up to get an AI match score and generate a tailored application in seconds.

Get your match score

Tags

Multi-Agent SystemsPythonNode.jsLLM IntegrationsRAGLLM EvaluationGCPWorkflow OrchestrationAI GovernanceVector Databases

About the role

Role Overview

Grafana Labs is hiring a Staff AI Engineer (fully remote) to own the AI agent infrastructure and automation platform powering its Marketing Operations organization. This is a high-autonomy role where you identify high-leverage problems, shape the technical direction, and deliver end-to-end multi-agent automation systems.

Key Missions / Responsibilities

  • Own technical direction for the AI agent infrastructure and automation platform; identify high-leverage problems and design solutions.
  • Lead end-to-end development of multi-agent AI systems, including:
    • Architecture and implementation
    • Testing, deployment, and ongoing production operations
  • Establish governance & compliance for AI workflows, including:
    • Access controls
    • Audit trails
    • PII handling
    • Human-in-the-loop escalation paths
  • Build multi-agent architectures and LLM-integrated automation, partnering with multiple teams to improve operational efficiency.

Requirements

  • Experience building and operating multi-agent systems at scale, including:
    • Agent decomposition
    • Orchestration patterns (sequential chains, router/dispatcher, parallel fan-out)
    • State management
    • Production monitoring
  • Strong technical communication with both engineers and business stakeholders.
  • Ability to identify high-leverage problems, push back on low-impact requests, and deliver end-to-end with minimal direction.
  • Hands-on experience with LLM frameworks and patterns, including:
    • Prompt engineering
    • RAG
    • Function calling / tool use
    • Structured output parsing
    • Evaluation
  • Strong understanding of LLM failure modes and production mitigations, including:
    • Confidence thresholds
    • Fallback logic
    • Human escalation
    • Cost/latency management
  • 2+ years applying LLMs/AI to production workflows (not only prototypes).
  • 8+ years software engineering experience, with depth in backend development, systems integration, or data/analytics engineering.
  • Strong proficiency in Python and JavaScript/Node.js, including Git-based workflows, code review, and testing discipline.
  • Ability to diagnose business problems and think in workflows/outcomes.
  • Fluency with AI-assisted development tools (e.g., GitHub Copilot, Cursor, Claude Code) and using AI to build AI systems.

Nice to Have / Additional Signals

  • Deep familiarity with GCP tools and services (e.g., BigQuery; Cloud Functions/Cloud Run; serverless/containerized services).
  • Experience with AI governance/compliance practices.
  • Active in open-source communities (Grafana is built on OSS).
  • Prior automation experience in marketing, sales, or customer success (B2B SaaS).
  • Familiarity with Model Context Protocol (MCP) or similar standards.
  • Experience with workflow orchestration tools (e.g., n8n, Temporal, Prefect, Airflow).
  • Observability tooling for AI systems (e.g., LangSmith, Weights & Biases).
  • Some familiarity with frontend frameworks for AI tool interfaces (e.g., React; Slack Block Kit).
  • Familiarity with marketing/sales platforms (e.g., Salesforce, HubSpot, Marketo, Outreach, Customer.io).
  • Vector databases / retrieval pipelines (e.g., Pinecone, Weaviate, ChromaDB, Qdrant, pgvector).

About Grafana Labs

Grafana Labs is an open-source-focused software company best known for Grafana, a leading observability platform. The company builds developer and operations tooling and runs teams that deliver infrastructure and automation products for customers across engineering and business workflows.

Scraped 5/12/2026