LLMs Track State Over 196 Tool Calls for MD5

Evaluating large language models (LLMs) on long-horizon tasks is challenging because errors can cascade catastrophically when each step depends on the previous one. Existing benchmarks often conflate state-tracking difficulty with instruction interpretation and are susceptible to shortcuts, making it hard to pinpoint why a long task fails. This research specifically isolates and tests an LLM's capacity to carry exact intermediate state across numerous tool calls. The study uses the computation of an MD5 cryptographic hash as a clean test case. This involves a sequence of 196 dependent tool calls over 64 rounds, where the LLM must carry four 32-bit words (a,b,c,d) in its context from one call to the next. The task's interpretation is simplified, and the MD5 implementation from scratch allows for bit-level verification, ensuring any failure is purely due to state bookkeeping. A mixture-of-experts model, gpt-oss-120b, with approximately 5.5B active parameters per token, successfully carried the full state across all 196 calls and returned the correct digest on a majority of completed runs. Key to its success were two factors: consistently keeping the model's own reasoning in its context each turn, and employing voting over a "thinking-enabled" worker LLM to correct modular arithmetic errors. The research also localizes residual failures, distinguishing between state-carrying issues, arithmetic errors, and serving problems.