New Benchmark Advances Theory-Scale Auto-Formalization for Computer Science.

Auto-formalization, the process of automatically translating natural language mathematics into formal proofs, is crucial for scaling formal verification efforts. However, existing methods primarily focus on isolated statements, leaving the challenge of coherently formalizing entire theories with hundreds of interdependent definitions and theorems largely unaddressed. This new work tackles this gap by introducing LCS-Bench. LCS-Bench is a novel, theory-scale benchmark specifically designed for Logics for Computer Science. It was constructed using an innovative semi-automated agentic pipeline that incorporates concept graphs, formal signature planning, issue tracking, and counter-example search, all reviewed by human experts for faithfulness. The resulting dataset is substantial, comprising 327 textbook items, over 4,076 Lean declarations, and more than 85,000 lines of Lean code. The benchmark supports extensive evaluation through a data engine that generates five distinct evaluation tracks, measuring various aspects of auto-formalization and theorem-proving capabilities. A new evaluation protocol with definitional equivalence checkers allows for more precise and faithful assessment. Initial evaluations of 14 models show that LCS-Bench is high-quality and challenging, with state-of-the-art models achieving only 20.1% on auto-formalization tasks, highlighting significant room for improvement in this critical area.