ModelEquivBench Certifies LLM-Generated Optimization Model Equivalence

Large language models (LLMs) are increasingly used to generate optimization models from natural language descriptions. However, current evaluation methods often provide only a binary "equivalent/not-equivalent" verdict or an execution success rate, which lacks independent checkability and fails to capture the nuanced ways models can agree or disagree. ModelEquivBench addresses this by introducing a certifying, multi-relational evaluation system. It generates a detailed semantic profile (E0-E6) for each pair of generated and ground-truth models, covering aspects from model construction and ingestion (E0) to optimal-value equality (E5) and optimizer-set equivalence (E6). Each entry in this profile comes with relation-appropriate, independently re-checkable evidence, such as replayable traces or exact-rational certificates. The system avoids guesswork by reporting typed UNKNOWN or N/A outcomes for incomplete mappings or resource limits. When applied to evaluate models like GPT-5.4, Claude Sonnet 4.6, and Qwen3.5-397B-A17B, ModelEquivBench revealed distinctions that coarse baselines missed. For example, many executable candidates were certified negative on specific relations, and structural rejections occurred even when feasible sets were equivalent, demonstrating the need for a more granular and verifiable evaluation approach.