TriRoute Unifies LLM Routing for Attention, Experts, and KV-Cache

Current methods for optimizing large language model inference often tackle individual components like Mixture-of-Experts (MoE) for FFN sparsification, Mixture-of-Depths (MoD) for block skipping, or KV-cache quantization for memory compression. However, these decisions are highly interdependent; a token requiring full attention might also need high-precision caching, regardless of the expert processing it. TriRoute addresses this by introducing a single, shared controller that coordinates these three critical decisions for every token at every layer. It determines the optimal attention mode (skip, local, full), selects a sparse set of FFN experts, and assigns an appropriate KV-cache bit-width. This end-to-end training, using a heterogeneous relaxation and a Lagrangian budget constraint, allows for a controllable knob over average compute and memory costs. The research demonstrates that TriRoute Pareto-dominates the best independent combinations of MoD, MoE, and KV-quantization. It achieves superior inference FLOPs and memory efficiency while crucially preserving robustness for tail-case scenarios like rare entities, code, and arithmetic, which pure perplexity optimization often erodes. The controller intelligently allocates resources, for instance, assigning full attention and high-precision cache to sentence-initial positions and named entities.