Chain-of-Thought Transformers Can Efficiently Simulate Complex Algorithms

The growing interest in "reasoning" models, specifically Chain-of-Thought (CoT) transformers, is partly due to their theoretical ability to simulate Turing machines and thus perform arbitrary computations. However, the Turing machine model is often impractical for analyzing algorithms, which are typically designed for the more efficient Word RAM model. This study addresses whether CoT transformers can efficiently simulate Word RAM algorithms, which operate with random-access memory and unit-cost operations on words. The researchers affirmatively show that CoT transformers can indeed simulate any Word RAM algorithm with only a poly-logarithmic overhead. This efficiency holds across different CoT configurations, including those with finite precision, continuous reasoning vectors, and hybrid architectures combining transformer layers with recurrent layers. The results highlight that CoT transformers can perform complex algorithmic tasks, such as sorting or running Dijkstra's algorithm, with efficiency comparable to traditional Word RAM models, a significant improvement over the quadratic overhead typically associated with Turing machine simulations.