Product Data Scientist (AI Evaluation & Quality)
Finom
full-remotemidpermanentdataother Full remote 135 days ago via WTTJ
See how well this job matches your profile
Sign up to get an AI match score and generate a tailored application in seconds.
Get your match scoreTags
PythonSQLLLM EvaluationAI Quality AnalyticsStatisticsHypothesis TestingSamplingDashboardsRAGLLM Agents
About the role
Role overview
Join Finom as a Product Data Scientist focused on AI evaluation and quality. You will own the evaluation lifecycle across products—measuring quality offline, monitoring it online, and using production feedback to continuously improve AI/LLM behavior.
Key missions
- Own and extend the offline evaluation suite across products, including datasets, judges, and metrics.
- Build and maintain online quality dashboards and monitor key indicators such as:
- resolution rate
- customer satisfaction rate
- Close the production feedback loop by analyzing real traffic to identify failure patterns and turn them into regression cases.
- Harden evaluation methodology and translate metrics into actionable decisions.
Requirements
- Statistics foundation (e.g., sampling, hypothesis testing, variance; knowing how to interpret noisy metrics).
- Python and SQL to build analysis end-to-end.
- 3+ years in analyst/data scientist roles, with at least one role in a product context.
- Quality analytics for ML systems, including areas such as ranking, recommendations, and classification.
- Hands-on experience evaluating LLM applications (e.g., RAG, agents, tool use, judges).
Nice-to-haves
- Experience building LLM agents (side projects, toy builds, or personal experiments are acceptable).
About Finom
Finom is a product company working on AI/ML systems and product analytics to improve product quality. The role focuses on evaluating AI performance, building quality dashboards, and turning production feedback into decision-making.
Scraped 7/10/2026