Principal Data Scientist (Agent Builder)

Role Overview

As Principal Data Scientist (Agent Builder) at Elastic, you will shape the technical direction of the agentic platform and define evaluation strategies that guide product decisions. You’ll lead work on frontier AI/ML problems and partner with engineering, product, UX, and other data scientists to turn advanced research into measurable improvements.

Key Responsibilities

• Define the evaluation strategy for conversational and agentic search, including:
  • Offline and online evaluation
  • Reference datasets
  • A/B testing
• Design quality metrics and decision frameworks for:
  • RAG, agents, tools
  • Model selection and agent routing
  • Prompt behavior
  • Cost/latency trade-offs
• Build, compare, and guide improvements in retrieval approaches, including:
  • Sparse and dense retrieval
  • Retrieval and re-ranking
  • Vector search
  • Query understanding and context enrichment

Requirements

• Experience influencing product/technical strategy with data in ambiguous or emerging domains
• 8+ years of applied DS/ML experience with deep expertise in at least several of:
  • Information Retrieval, NLP, ranking, semantic search
  • RAG and/or LLM-powered product experiences
• Proven experience defining and leading evaluation for production AI/ML systems, including:
  • Offline metrics and online experimentation
  • LLM-as-judge approaches
  • Groundedness and citation quality
  • Model comparison
• Practical experience with Elasticsearch or similar search/distributed data systems (ES|QL familiarity is a plus)
• Hands-on ability with:
  • Python, PyTorch/Transformers, Pandas
  • Notebooks, reproducible experiments, versioned datasets
  • Clean, reviewable code
• Collaboration with engineering to move from prototype to production, including:
  • Telemetry design and dashboards
  • CI guardrails
  • Quality regression tracking
• Strong communication and mentoring skills; ability to communicate trade-offs to technical and business stakeholders

Nice to Have

• ES|QL familiarity