TREAT Benchmark Tests LLM Mathematical Knowledge Robustness

AI systems frequently need to bridge the gap between flexible input formats and the precise formal objects used by downstream tools. A significant challenge arises when these systems must identify a known formal object even when it's presented in an unfamiliar, yet mathematically equivalent, formulation. This research explores this problem through "theorem recognition," where a model must recover a theorem's identity from a transformed version of its condition. To assess this, the TREAT benchmark was developed. Unlike benchmarks that paraphrase theorem text, TREAT modifies the mathematical form of theorem conditions themselves, presenting known results through various equivalent expressions like residual equations, optimization identities, or set relations. The corpus comprises 737 theorem identities and nearly 30,000 transformed variants, generated from scraped theorem pages. Evaluations on a panel of models showed that even the best model could only retrieve the correct theorem identity in about 60% of cases. Other models exhibited different failure modes, including abstention or generating malformed outputs. These results suggest that LLMs' knowledge of theorems is surprisingly fragile when faced with equivalent changes in mathematical representation, highlighting a critical area for improvement in AI systems that require robust access to formal knowledge.