MUX Enables Efficient, Continuous Reasoning in Language Models

Large language models (LLMs) typically solve complex problems by generating explicit, step-by-step reasoning in natural language. While effective, this process is computationally intensive because each reasoning step often conveys minimal information and much of the computation is spent on expressing thoughts rather than performing calculations. A new method, MUX, addresses this by enabling high-bandwidth and compact reasoning. MUX achieves this by distilling discrete reasoning into continuous, multiplexed tokens within a latent space. Each latent token is trained to represent a weighted linear superposition of a span of discrete reasoning subwords, designed to be lossless, meaning the original span can be fully recovered. The paper proves that specific position-dependent weightings, such as geometric decay, support this lossless multiplexing, preventing issues like latent collapse. MUX also facilitates parallel exploration in search-intensive problems. Across 32 evaluation settings and four LLMs, MUX consistently outperformed existing latent reasoning baselines, with analyses showing that the learned latent tokens encode faithful and interpretable reasoning.