Reasoning Jury Improves LLM Evaluation with Multi-Model Consensus

Evaluating the quality of long reasoning traces from Large Language Models (LLMs) is crucial for data curation, reinforcement learning signals, and understanding model behavior. However, single-model judges, even frontier ones, often struggle to accurately identify reasoning defects. This research introduces Reasoning Jury, a novel system that replaces a single judge with a panel of LLMs and a moderated consensus mechanism. In Reasoning Jury, defects and their severity are identified through a deliberation process where jurors critique each other's judgments, modifying their initial votes. A moderator then derives a consensus. This approach allows a jury of open-weight models (e.g., gpt-oss-120b) to significantly outperform frontier models like Opus-4.6 and Gemini-3.1-Pro in correctly identifying reasoning defects. Beyond accuracy improvements, the aggregated cost of the Reasoning Jury, including initial verdicts and deliberations, is only 8% to 15% of the cost of running frontier models in a typical LLM-as-a-judge setup. The system also provides deeper insights into LLM failure modes on benchmarks, enhancing understanding of model performance.