Key Responsibilities

• Design and implement evaluation frameworks for multi-agent AI systems
• Lead experimentation and A/B testing initiatives
• Curate and manage datasets for model training and evaluation
• Ensure reliability and safety of agent workflows in production-like environments
• Build and maintain data pipelines and experiment tracking workflows
• Monitor system reliability and performance using Azure Monitor and Application Insights
• Collaborate with cross-functional teams on LLM and agentic system development

Requirements

• 8+ years of relevant experience in data science or machine learning
• Proficiency in Python and ML frameworks
• Hands-on experience with LLM APIs and agentic frameworks such as LangChain, LlamaIndex, and Semantic Kernel
• Familiarity with evaluation tooling including Ragas, DeepEval, LangSmith, or similar
• Strong foundation in statistics, NLP, prompt engineering, and A/B testing methodology
• Experience with data pipelines, experiment tracking (MLflow, W&B), and CI/CD for ML workflows
• Proficiency in Azure ML, Azure OpenAI Service, and Azure AI Foundry
• Familiarity with Azure Monitor and Application Insights for tracking reliability and performance
• Immediate availability required

Skills Required