MIRA-Math Benchmarks LLM Information Requesting and Math Reasoning

Existing mathematical reasoning benchmarks for Large Language Models typically provide all necessary information upfront, while interactive benchmarks often conflate reasoning with tool use, retrieval, and complex dialogue. A new benchmark, MIRA-Math, has been introduced to specifically test a narrower, yet crucial, diagnostic capability: solving math problems where only one essential piece of information is missing. In MIRA-Math, an LLM solver must identify the single missing atomic fact, request it in natural language within a budget, and then integrate the provided fact to arrive at an exact final answer. A constrained LLM responder provides the exact fact only if the request matches precisely, ensuring deterministic validation. This setup allows for precise measurement of information requesting success and subsequent computational accuracy. The benchmark comprises 2,310 generated instances across 22 mathematical families, including algebra, probability, and discrete structures. Initial experiments with various models show that the ability to correctly request information is distinct from the ability to perform the downstream calculation, highlighting separate areas for model improvement. The release includes generators, verifiers, and prompts to foster reproducible research in this area.