Evaluating Faithful Formalization of Natural Language to Lean Statements

A new study investigates the fidelity of converting natural language mathematical statements into formal Lean declarations, moving beyond simple compilation checks. The researchers developed a rigorous evaluation protocol that combines Lean compilation, cross-model semantic judging, and human expert calibration to assess "faithful statement formalization." Using a 400-entry benchmark covering advanced mathematics, the study uncovered a substantial discrepancy: while a tool-augmented agent achieved an 89.5% compilation rate, only 60.5% of these outputs were deemed semantically faithful. This 29.0-point gap indicates that a statement can be syntactically correct in Lean but still misrepresent the original natural language meaning. The research also analyzed the impact of various interventions, finding that Lean elaboration feedback is crucial for validity but can expose more semantic failures. Context search improves grounding, and expert drafting can be substituted when feedback and grounding are strong. The findings emphasize the need to separately report formal validity, proof-oriented competence, and faithful statement generation.