Efficient, Privacy-Aware Edge-Cloud LLM Inference Framework Introduced

A new framework has been developed to enable more efficient and private inference for large language models (LLMs) by distributing computational tasks between edge devices and cloud infrastructure. This approach tackles the common challenges of high latency, limited hardware resources on devices, and critical user privacy concerns. The system intelligently splits the LLM inference process, with local endpoints handling tasks like input preprocessing, embedding, and speculative decoding, while the cloud manages authenticated decoder inference and KV cache. All data transmitted between the edge and cloud is quantized and encrypted using AES-GCM, ensuring sensitive user information remains protected. The framework incorporates an endpoint-authenticated KV cache, which is crucial for maintaining privacy and security during collaborative inference. Key lightweight modules, draft parameters, and cache access policies are kept local to prevent data leakage. This design supports a variety of heterogeneous devices, from CPU-only to GPU-equipped and embedded systems, through optimized streaming and batching. Evaluations show that this collaborative inference method significantly reduces per-token latency by up to 46.1% and decreases downlink data payloads by up to 67.4% compared to existing split inference baselines. Importantly, it achieves performance comparable to full cloud inference, making advanced LLM capabilities more accessible and secure on consumer and edge devices.