LLM-as-a-Judge Framework Validated for Agentic AI in Drug Discovery

Evaluating the open-ended outputs of agentic AI systems, particularly in complex scientific fields like drug discovery, presents a significant challenge. Traditional metrics often fail to capture semantic correctness, and human expert evaluation is not scalable. A new study introduces an "LLM-as-a-Judge" evaluation framework designed for AstraZeneca's agentic drug discovery assistant, ChatInvent, aiming to bridge this gap. The framework defines four key dimensions for evaluating output quality: Completeness, Relevancy, Structural Clarity, and Scope Adherence, alongside deterministic checks for tool call correctness. A crucial aspect of this work involved validating the LLM judge's alignment with human experts. Five expert annotators compared various LLMs (Gemini 3.1 Pro, Claude Opus 4.7, GPT-5, Llama 3.1 70B) as judges. The best-performing LLM judge was further optimized using few-shot demonstrations of human-annotated examples, which improved its alignment with the human majority vote from 0.80 to 0.86. Applying this optimized judge to held-out questions revealed specific limitations and surprisingly, found that informal phrasings did not degrade output quality, and even suggested that having the LLM rewrite the original question could be beneficial. This framework provides a robust, human-aligned template for evaluating agentic systems in scientific domains.