Compressing Recursive Reasoners for Edge Devices: Challenges and Solutions

Deploying recursive reasoning models, which solve complex structured tasks with few parameters by repeatedly updating a latent state, onto edge hardware presents significant compression challenges. Unlike conventional sequence models, where quantization errors accumulate across output tokens, in recursive models, these errors compound across multiple reasoning cycles. This study investigates the effects of aggressive compression on recursive reasoners, finding that while local prediction accuracy is maintained, global reasoning capabilities are severely compromised. For instance, naive INT4 pruning, distillation, and linear attention caused puzzle-exact accuracy to collapse to zero, even as cell accuracy held. This collapse is architectural, affecting MLP-mixing recursion but not attention-based models on the same task. To counteract this, the researchers successfully reversed the damage using per-channel calibrated INT4 quantization without retraining. They also introduced "carry-trajectory fidelity," a label-free signal (cosine similarity to the full-precision reasoning path) that accurately predicts this damage and its recovery before task evaluation. The resulting deployment recipe includes flash-streamed embeddings to remove a 99.4MB bottleneck, INT8 at one cycle matching full-depth accuracy at 6x fewer FLOPs, and calibrated INT4 fitting a 4MB microcontroller, enabling efficient edge deployment.