**This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for an Applied Data Scientist, Health AI Evaluation & Datasets based in the United States.** This role sits at the intersection of clinical expertise, data science, and cutting-edge AI evaluation, focusing on building trustworthy datasets for healthcare generative AI systems. You will design and validate the data foundations that power high-stakes models used in clinical, payer, pharmaceutical, and patient-facing environments. Working in a multidisciplinary team alongside engineers, researchers, and clinical experts, you will translate complex healthcare workflows into structured, measurable datasets and rigorous evaluation frameworks. This is a highly impactful position where your work directly influences the safety, accuracy, and reliability of health AI systems operating at scale. The environment is fast-moving, collaborative, and deeply focused on clinical realism, regulatory alignment, and measurable model performance. ### Accountabilities - Design, define, and operationalize high-quality healthcare datasets used for training, fine-tuning, and evaluating generative and multimodal AI systems across clinical and life sciences domains. - Translate complex healthcare objectives such as diagnosis support, medical summarization, RAG-based retrieval, and patient communication into structured dataset specifications, labeling schemas, and evaluation rubrics. - Develop clinically grounded evaluation frameworks that measure model performance across safety, accuracy, faithfulness, guideline adherence, and workflow relevance. - Design multimodal datasets spanning clinical notes, imaging, structured EHR data, claims, literature, and patient-provider communications while ensuring clinical validity and statistical rigor. - Define sampling strategies, annotation guidelines, SME review workflows, inter-annotator agreement standards, and quality assurance processes for healthcare datasets. - Build statistical and ML-based quality checks including bias analysis, subgroup performance evaluation, leakage detection, and dataset reliability metrics. - Collaborate with engineers and research scientists to integrate datasets into evaluation pipelines, including LLM-as-judge systems, benchmarking frameworks, and model comparison workflows. - Ensure data governance, compliance, and auditability across PHI-sensitive workflows, including de-identification, provenance tracking, and version control. - Evaluate model behavior beyond accuracy, including calibration, hallucination risk, safety-critical failure modes, and fairness across patient populations. - Support client-facing discussions by translating technical methodology into clear, defensible explanations for clinical and ML stakeholders. - Contribute to reusable internal assets such as taxonomies, rubrics, evaluation templates, and gold-standard datasets. **Requirements** - 5+ years of experience in data science, including at least 2+ years working directly with healthcare, biomedical, clinical, payer, pharma, or life sciences data. - Strong understanding of healthcare data systems and standards, including EHR structures and clinical coding systems such as ICD-10, CPT, SNOMED CT, LOINC, and RxNorm. - Proven experience designing and building ML datasets, including annotation guidelines, sampling strategies, QA processes, and dataset validation frameworks. - Hands-on experience with LLM-based AI workflows, including evaluation design, prompt engineering, retrieval-augmented systems, and rubric-based assessment methods. - Strong programming skills in Python and SQL, with familiarity in tools such as pandas, scikit-learn, statsmodels, and modern ML/LLM ecosystems. - Solid statistical background covering sampling methods, bias and fairness analysis, inter-annotator agreement, hypothesis testing, and uncertainty estimation. - Deep understanding of healthcare privacy and compliance frameworks such as HIPAA, de-identification methodologies, and secure handling of sensitive data. - Ability to collaborate effectively with clinicians, engineers, researchers, and business stakeholders in complex, cross-functional environments. - Strong communication skills with the ability to translate clinical and technical complexity into actionable insights and structured evaluation frameworks. - Advanced degree in a relevant field (biostatistics, epidemiology, health informatics, data science, computer science, or related discipline) or equivalent experience; clinical backgrounds (MD, RN, PharmD, MPH, PhD) are a strong advantage. **Benefits** - Competitive annual salary ranging from **$150,000 to $175,000 USD**, based on experience and qualifications. - Fully remote work opportunity across the United States. - Opportunity to work on high-impact healthcare AI systems used across clinical, payer, and life sciences domains. - Collaborative environment working alongside leading experts in AI, clinical science, and data engineering. - Exposure to cutting-edge generative AI, evaluation frameworks, and multimodal dataset development. - Professional growth in a rapidly evolving field at the intersection of healthcare and artificial intelligence. - Support for continuous learning in advanced ML, healthcare data systems, and AI evaluation methodologies. **How Jobgether works:** We use an **AI-powered matching process** to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team. We appreciate your interest and wish you the best! [Why Apply Through Jobgether?](https://jobgether.com/how-jobgether-works) **Data Privacy Notice:** By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time. #LI-CL1