Benchmarking Agentic AI Systems for Academic Peer Review

The rise of AI-assisted research is placing increasing pressure on traditional peer review systems, leading to the emergence of agentic AI review systems as a potential solution. However, a clear methodology for evaluating these new systems has been lacking. This research addresses this gap by benchmarking several open-source (OpenAIReview, coarse) and proprietary (Reviewer3) AI review systems, alongside a zero-shot baseline, across six different Large Language Models (LLMs). The study first assessed whether AI reviews align with paper quality, using external signals like citations and acceptance decisions for ICLR/NeurIPS papers. All systems performed above chance, with OpenAIReview combined with GPT-5.5 achieving the highest pairwise accuracy at 83.0%. Secondly, to test error detection, a perturbation benchmark was created by injecting four types of errors into papers across eight arXiv subject classes. The strongest configuration (OpenAIReview + GPT-5.5) detected 71.6% of these errors, indicating significant potential but also room for improvement. Interestingly, the union of detections across all six models reached 83.3% recall, suggesting that different models excel at detecting different types of errors and that improved system design could further boost performance. A public deployment of OpenAIReview also showed positive user feedback, with common complaints revolving around false positives and minor nitpicks. Overall, the findings indicate that AI review systems can effectively track human quality judgments and identify important errors, despite still having areas for refinement.