New Benchmark Evaluates LLM Performance on Combinatorial Code Synthesis
Summary
FindStatBench is a new execution benchmark designed to assess large language models' ability to synthesize combinatorial code, featuring 2,329 tasks and 5.52 million hidden instances. Evaluations of eleven systems reveal that top open and closed-source models perform similarly, examples can sometimes hinder performance, and output budget limitations can cause failures.
Why it matters
For professionals developing or integrating LLMs for code generation, this benchmark highlights current capabilities and limitations, especially in complex, constraint-heavy domains like combinatorial mathematics.
How to implement this in your domain
- 1Consult FindStatBench results when selecting LLMs for code synthesis tasks.
- 2Design prompts carefully, considering that examples might sometimes degrade performance for certain problem types.
- 3Account for potential output budget limitations in LLM applications requiring extensive reasoning.
- 4Prioritize LLMs that demonstrate stronger performance on map synthesis if that is a critical use case.
Who benefits
Key takeaways
- FindStatBench is a new benchmark for evaluating LLMs on combinatorial code synthesis.
- Top open and closed-source LLMs show similar performance on this benchmark.
- Providing input-output examples can sometimes surprisingly hinder LLM performance.
- Output budget constraints can prevent LLMs from completing complex reasoning tasks.
Original post by Soham Dan
"arXiv:2607.18260v1 Announce Type: new Abstract: We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis. Built from FindStat, it contains 2,329 tasks across 24 collections and 5.52M hidden instances, covering statist…"
View on XOriginally posted by Soham Dan on X · view source
Want to go deeper?
Turn these trends into skills with Learnijoy's hands-on AI & tech courses.
Explore coursesMore in AI Research
New Tool Generates Contamination-Resistant, Labeled Code Datasets for LLMs
Spaghetti Architect is a new open-source tool that generates controlled, multi-language code datasets, addressing issues of contamination and lack of semantic control in existing code corpora. It creates correct-by-construction programs with adjustable "messiness" and difficulty labels, making it ideal for training and evaluating code-generating LLMs.
New Method Safely Gates Hazardous LLM Knowledge Without Deletion
Researchers introduce Token Inoculation, a method that allows large language models to retain sensitive "dual-use" knowledge while selectively refusing hazardous queries. This approach uses a special token to condition the model's behavior, improving safety without sacrificing benign domain performance.
GNNAS-TSP Selects Optimal Algorithms for Traveling Salesman Problem
Researchers introduce GNNAS-TSP, a Graph Neural Network (GNN)-based framework for automated algorithm selection (AS) for the Traveling Salesman Problem (TSP). GNNAS-TSP learns TSP instance representations directly from raw graph data, avoiding manual feature engineering, and formulates AS as a joint cost-prediction and ranking task to select the best solver from a portfolio under fixed computational budgets.