Dual-Flow Transformers Optimize LLM Inference by Decoupling Prefill and Decode.

A new transformer architecture, the Dual-Flow Transformer, has been proposed to optimize the inference costs of large language models by decoupling the prompt prefill and autoregressive decode phases. These two phases have distinct hardware demands: prefill is parallel and compute-bound, while decode is sequential and memory-bandwidth-bound. Traditional scaling methods increase costs for both phases simultaneously. The Dual-Flow Transformer features a primary flow that handles prompt processing and populates the key-value (KV) cache, and an auxiliary flow activated only during continuation prediction. This auxiliary flow adds computation without writing persistent state or affecting the primary flow. Both flows share major attention, MLP, and output matrices, enabling weight reuse and cached key/value reuse during grouped execution. Experiments show that Dual-Flow Transformers achieve lower validation loss across various configurations. In Mixture-of-Experts (MoE) models, this separation allows independent control over prefill and continuation costs, demonstrating a clear trade-off between prefill, decode, and predictive quality.