Mask-Proof Pipeline Automates LLM Evaluation for Mathematical Proofs

Researchers have developed Mask-Proof, an innovative pipeline designed to automate the curation and evaluation of mathematical proofs using large language models (LLMs). The primary goal is to provide a scalable and reproducible method for measuring step-level reasoning within long and diverse mathematical proofs, addressing a critical gap in evaluating trustworthy AI assistance in scientific progress. The Mask-Proof pipeline transforms actual mathematical proofs into masked-step tasks that can be automatically checked. It strategically masks key formula steps within a proof, provides the surrounding context, and then evaluates how well an LLM can reconstruct the missing steps. The reconstruction quality is judged by another LLM-based equivalence judge, which uses repeated votes to ensure stability and accuracy. Accompanying the pipeline is Mask-ProofBench, a new benchmark comprising 292 curated problems from various research areas. Experiments conducted with 17 different models on this benchmark revealed that models enhanced with specific reasoning capabilities significantly outperform standard models, showing improvements of 12% to 27%. The LLM-based evaluator achieved a high agreement rate of 96.8% with expert annotators, confirming its reliability for faithful, reproducible, and comparable measurement of step-level mathematical reasoning.