Data Scientist (Evals, Member of Technical Staff)
Perplexity AI
full-remoteseniorpermanentdatabackend Full remote 144 days ago via WTTJ
See how well this job matches your profile
Sign up to get an AI match score and generate a tailored application in seconds.
Get your match scoreTags
PythonSQLAWSDatabricksLLM-as-a-judgeEvaluation PipelinesGround Truth DatasetsLLMs at ScaleInformation RetrievalVLM
About the role
Role Overview
Join Perplexity as a Data Scientist to build and maintain automated evaluation pipelines that assess answer quality across the company’s products. You’ll design evaluation sets and methods, develop VLM-based approaches, and continuously incorporate insights from public benchmarks and academic evaluations.
Responsibilities
- Architect and maintain automated evaluation pipelines to measure answer quality across Perplexity products.
- Design evaluation sets and methods to assess how tool calls impact final answer quality.
- Regularly review and adapt public benchmarks and academic evaluations into ongoing performance measurement.
- Collaborate with technical leadership within a small, high-impact team to measure and improve answer quality.
- Define and operationalize evaluation metrics (e.g., factual consistency, hallucination rate, retrieval precision) and build ground truth datasets.
Requirements
- 4+ years of experience in Data Science or Machine Learning.
- 1+ years working with LLMs at scale, specifically LLM-as-a-judge setups.
- Strong Python and SQL skills, able to write production-grade code.
- Experience building within a modern cloud data stack, specifically AWS and Databricks.
- PhD or MS in a technical field, or equivalent experience.
- Comfort with agentic coding workflows and AI-assisted development tools.
- Research background applying research methods to real-world ML problems.
- Experience working on customer-facing web products/consumer apps with real user traffic.
Nice-to-Haves
- Experience evaluating tool-call impact explicitly (as part of evaluation design).
- Experience with VLM-based solutions.
About Perplexity AI
Perplexity AI builds AI-powered products that generate answers using large language models. The company focuses on improving answer quality through evaluation and continuous measurement across its offerings.
Scraped 7/1/2026