TriRoute Unifies Adaptive Attention, Experts, and KV-Cache Allocation

Conditional computation methods aim to improve language model efficiency by decoupling quality from per-token inference cost, but existing techniques often optimize only one aspect in isolation, such as Mixture-of-Experts (MoE) for FFN sparsification, Mixture-of-Depths (MoD) for block skipping, or KV-cache quantization for memory compression. This paper argues that these three decisions—attention resolution, expert selection, and cache bit-width—are strongly coupled and should be made together. Researchers introduce TriRoute, a single, lightweight controller that operates across all three axes. For each token at every layer, TriRoute emits a coordinated policy, determining the attention mode (skip, local, or full), a sparse set of FFN experts (including a null expert for MoD), and the KV-cache bit-width. The controller is trained end-to-end using a heterogeneous relaxation under a Lagrangian budget constraint, allowing the average compute and memory cost to be a controllable parameter. The paper identifies and addresses a cross-axis routing-collapse cascade that can occur in naive joint training, using per-axis normalization and a coupling-aware balancing loss. Evaluations on decoder-only models demonstrate that TriRoute Pareto-dominates the best independent combinations of MoD, MoE, and KV-quantization at matched inference FLOPs and memory, while also preserving robustness on rare entities, code, and arithmetic. Post-hoc analysis reveals that the controller intelligently allocates full attention and high-precision cache to critical elements like sentence-initial positions and named entities, while routing function words more cheaply.