GPT-5.6 Sol Performance Triples with API Settings Adjustment

@OpenAI· July 29, 2026 View original

Summary

GPT-5.6 Sol, despite solving math problems, struggled with the ARC-AGI-3 2D puzzle benchmark until researchers enabled "retained reasoning" and "context compaction" API settings, tripling its score with fewer tokens.

Researchers investigated why GPT-5.6 Sol, a model capable of solving complex mathematical problems, performed poorly on the ARC-AGI-3 benchmark, which tests a model's ability to learn unfamiliar 2D games without explicit instructions. They discovered that the standard evaluation setup was hindering the model's performance by discarding its reasoning after each move and dropping earlier actions as context filled up, forcing it to repeatedly start over. By enabling two specific API settings—"retained reasoning" and "context compaction"—within their custom harness using the Responses API, the model's score on the public ARC-AGI-3 set increased by 188%, while simultaneously reducing output token usage by six times. This highlights that benchmark scores are not solely a measure of the model itself but also reflect the influence of API settings, harness design, and prompting strategies. Developers are advised to use these optimized settings for maximizing performance in long-running agent applications.

Why it matters

This research demonstrates that effective deployment and evaluation of advanced AI models heavily depend on API configurations and harness design, not just the model's inherent capabilities. Professionals need to understand these nuances to maximize model performance and interpret benchmarks accurately.

How to implement this in your domain

1Review current LLM integration strategies to ensure optimal API settings are being utilized.
2Experiment with "retained reasoning" and "context compaction" for agentic or long-running AI applications.
3Evaluate existing AI benchmarks and internal testing harnesses for potential biases introduced by API settings.
4Prioritize using the latest API versions (e.g., Responses API over legacy Chat Completions API) for better features.
5Train development teams on best practices for prompt engineering and API configuration to enhance model performance.

Who benefits

AI DevelopmentSoftware EngineeringResearch & DevelopmentGaming

Key takeaways

API settings and harness design significantly impact LLM performance in benchmarks.
"Retained reasoning" and "context compaction" can dramatically improve agentic model capabilities.
Benchmark scores reflect both the model and the evaluation environment.
Optimizing API usage can lead to better results and reduced token costs.

Original post by @OpenAI

"GPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games? We investigated. The harness was not letting it remember what it had learned. We found that enabling two API settings tripled our scores with…"

View on X