Topology-Aware Data Movement for Disaggregated GPU Inference

This paper addresses a critical datacenter networking challenge arising from disaggregated Large Language Model (LLM) inference, where prefill and decode operations run on separate GPU pools. The primary issue is the massive data transfer required for the KV cache, which can exceed 100 GB/s aggregate for a 70B model at production scale. Existing systems often use uniform RDMA, failing to account for the vast differences in bandwidth across various physical interconnects (e.g., NVLink, InfiniBand, TCP). The researchers propose a topology-aware transfer orchestrator that discovers the interconnect hierarchy at startup and selects the optimal transport for each data transfer. This orchestrator employs three key mechanisms: pipelined layer-by-layer transfer to overlap transmission with computation, NVLink domain-aware placement for Mixture-of-Experts models to optimize KV cache locality, and CXL 3.0 memory expanders as a shared overflow tier for increased capacity and lower latency. While full evaluation on heterogeneous multi-node clusters with CXL 3.0 hardware is beyond current academic resources, analytical models and component implementations project a significant 3 to 18 times reduction in transfer latency compared to uniform RDMA.