Expert Review Reveals LLM Autoformalization Weaknesses Beyond Proof Gaps

This research investigates the distinction between merely closing proof gaps in interactive theorem provers and creating truly reusable library contributions through semi-autonomous formalization. The study focuses on a detailed case study involving the formalization of Grothendieck's vanishing theorem. Initially, the LLM-generated formalization compiled without any "sorries" (unproven statements), suggesting a complete proof. However, a subsequent expert review uncovered significant issues not related to proof correctness but to the quality and design of the formalization itself. These problems included suboptimal definitions, insufficient theorem generality, poor file organization, and an inadequate API. Following a review-driven refactoring and compression process, a second expert review was conducted. The comparison between the initial and refined versions clearly demonstrated that AI agents are adept at responding to local, mechanically verifiable feedback. However, they remain notably weak in making high-level design choices, such as selecting appropriate definitions and designing effective APIs for formal mathematical libraries. The study concludes that autoformalization should be judged not just by the absence of proof gaps, but by its ability to withstand rigorous expert review for reusability and quality.