Topology-Aware Data Movement Boosts Disaggregated LLM Inference

Disaggregated LLM inference, where prefill and decode stages run on separate GPU pools, introduces a significant datacenter networking challenge. The KV cache, which can be 2.6 GB per request for a 70B model, needs to be transferred between these pools at aggregate speeds exceeding 100 GB/s. Existing systems often use uniform RDMA, failing to account for the vast differences in bandwidth (up to 72x) depending on the physical relationship between GPUs, such as NVLink within a domain versus InfiniBand across nodes. This research designs a topology-aware transfer orchestrator to address this. The orchestrator discovers the interconnect hierarchy at startup and selects the optimal transport method for each data transfer. It employs three key mechanisms: first, pipelined layer-by-layer transfer, which overlaps transmission with ongoing prefill, effectively hiding 60% to 85% of latency behind computation. Second, NVLink domain-aware placement is used for Mixture-of-Experts models, co-optimizing expert dispatch with KV cache locality. Third, CXL 3.0 memory expanders are utilized as a shared overflow tier, offering 6x capacity at 86x lower latency than NVMe. While full evaluation on multi-node clusters with heterogeneous interconnects and CXL 3.0 hardware is beyond current academic resources, analytical bandwidth models, component implementations, and projected analysis across three architectures demonstrate a 3 to 18x reduction in transfer latency compared to uniform RDMA.