xelys jobs xelys jobs

Data Scientist (Evals, Member of Technical Staff)

Perplexity AI

full-remoteseniorpermanentdatabackend Full remote 144 days ago via WTTJ

See how well this job matches your profile

Sign up to get an AI match score and generate a tailored application in seconds.

Get your match score

Tags

PythonSQLAWSDatabricksLLM-as-a-judgeEvaluation PipelinesGround Truth DatasetsLLMs at ScaleInformation RetrievalVLM

About the role

Role Overview

Join Perplexity as a Data Scientist to build and maintain automated evaluation pipelines that assess answer quality across the company’s products. You’ll design evaluation sets and methods, develop VLM-based approaches, and continuously incorporate insights from public benchmarks and academic evaluations.

Responsibilities

  • Architect and maintain automated evaluation pipelines to measure answer quality across Perplexity products.
  • Design evaluation sets and methods to assess how tool calls impact final answer quality.
  • Regularly review and adapt public benchmarks and academic evaluations into ongoing performance measurement.
  • Collaborate with technical leadership within a small, high-impact team to measure and improve answer quality.
  • Define and operationalize evaluation metrics (e.g., factual consistency, hallucination rate, retrieval precision) and build ground truth datasets.

Requirements

  • 4+ years of experience in Data Science or Machine Learning.
  • 1+ years working with LLMs at scale, specifically LLM-as-a-judge setups.
  • Strong Python and SQL skills, able to write production-grade code.
  • Experience building within a modern cloud data stack, specifically AWS and Databricks.
  • PhD or MS in a technical field, or equivalent experience.
  • Comfort with agentic coding workflows and AI-assisted development tools.
  • Research background applying research methods to real-world ML problems.
  • Experience working on customer-facing web products/consumer apps with real user traffic.

Nice-to-Haves

  • Experience evaluating tool-call impact explicitly (as part of evaluation design).
  • Experience with VLM-based solutions.

About Perplexity AI

Perplexity AI builds AI-powered products that generate answers using large language models. The company focuses on improving answer quality through evaluation and continuous measurement across its offerings.

Scraped 7/1/2026