xelys jobs xelys jobs

Product Data Scientist (AI Evaluation & Quality)

Finom

full-remotemidpermanentdataother Full remote 135 days ago via WTTJ

See how well this job matches your profile

Sign up to get an AI match score and generate a tailored application in seconds.

Get your match score

Tags

PythonSQLLLM EvaluationAI Quality AnalyticsStatisticsHypothesis TestingSamplingDashboardsRAGLLM Agents

About the role

Role overview

Join Finom as a Product Data Scientist focused on AI evaluation and quality. You will own the evaluation lifecycle across products—measuring quality offline, monitoring it online, and using production feedback to continuously improve AI/LLM behavior.

Key missions

  • Own and extend the offline evaluation suite across products, including datasets, judges, and metrics.
  • Build and maintain online quality dashboards and monitor key indicators such as:
    • resolution rate
    • customer satisfaction rate
  • Close the production feedback loop by analyzing real traffic to identify failure patterns and turn them into regression cases.
  • Harden evaluation methodology and translate metrics into actionable decisions.

Requirements

  • Statistics foundation (e.g., sampling, hypothesis testing, variance; knowing how to interpret noisy metrics).
  • Python and SQL to build analysis end-to-end.
  • 3+ years in analyst/data scientist roles, with at least one role in a product context.
  • Quality analytics for ML systems, including areas such as ranking, recommendations, and classification.
  • Hands-on experience evaluating LLM applications (e.g., RAG, agents, tool use, judges).

Nice-to-haves

  • Experience building LLM agents (side projects, toy builds, or personal experiments are acceptable).

About Finom

Finom is a product company working on AI/ML systems and product analytics to improve product quality. The role focuses on evaluating AI performance, building quality dashboards, and turning production feedback into decision-making.

Scraped 7/10/2026