Clinical Annotator

Role Overview

Clinical Annotator (Remote / Work from home in the USA) You will act as a clinical Human Subject Matter Expert (SME) to validate pre-annotated clinical data, refine annotation guidelines, and help develop high-quality golden datasets for NLP model training and evaluation.

Responsibilities

• Perform rigorous human validation on pre-annotated data produced by clinical NLP models (e.g., Amazon Comprehend Medical) or internal LLM/encoder-based tools.
• Iteratively refine annotation guidelines and examples to improve inter-annotator agreement and overall dataset quality.
• Resolve disagreements between pre-annotations and human validation to ensure accuracy and consistency.
• Collaborate with Data Science and NLP teams by providing feedback on model performance and contributing to continuous improvement of pre-annotation models.
• Support creation and maintenance of golden datasets for training and evaluation.
• Participate in discussions on data sensitivity and ensure compliance with data privacy and security protocols, especially for patient data.
• Work across diverse clinical annotation projects, including:
  • Alcohol and smoking use cases
  • Oncology (e.g., ECOG, Karnofsky scores)
  • Biomarkers
  • SDO
  • Mental health

Requirements

Must Have

• Bachelor’s or Master’s in a relevant field (Life Sciences, Linguistics, Computer Science, or related healthcare discipline).
• Proven experience in data annotation for clinical/biomedical text.
• Strong foundational understanding of NLP concepts and terminology.
• Exceptional attention to detail and ability to maintain high accuracy.
• Strong communication and collaboration skills.
• Ability to quickly adapt to annotation tools such as Inception, Label Studio, or Prodigy.

Good To Have

• Experience with the John Snow Labs (JSL) ecosystem, including Health AI Lab and GenAI tools.
• Experience with advanced annotation types:
  • NER (Named Entity Recognition)
  • Relationship Extraction
  • Assertion status annotation
• Direct clinical background or extensive practical experience handling sensitive patient data.
• Familiarity with incremental batch training processes for machine learning.