CONCORD Boosts RAG Throughput in Device-Cloud Private Data Settings

Retrieval-augmented generation (RAG) is a key technique for enhancing language models by integrating external knowledge. A new challenge arises with device-cloud collaborative inference, where smaller language models on edge devices need to access private documents locally while leveraging public knowledge from the cloud. Privacy and policy often prevent direct document exchange, creating a "document-isolated dual-end RAG" environment. Existing RAG methods in this setting suffer from high latency and bandwidth usage due to frequent synchronization and dense evidence transfer. To overcome this, researchers propose CONCORD, an asynchronous sparse aggregation framework. CONCORD treats the cloud as an asynchronous evidence source, employing "waiting debt control" to decide when to wait for remote input and a "certificate-guided minimal supplementation" mechanism to request only essential remote evidence. This approach allows many decoding steps to commit locally without remote evidence, while still preserving the greedy token decisions of dense dual-end aggregation when consulting the cloud. Experiments show CONCORD significantly improves end-to-end throughput (1.66x to 2.15x) and reduces per-token communication by over two orders of magnitude, all while maintaining comparable answer quality.