SPADE: Speculative Decoding for Efficient Distributed LLM Inference.

A new distributed inference framework, named SPADE, has been developed to address the high computational costs and deployment challenges associated with large language models (LLMs). SPADE innovatively integrates speculative decoding across both edge and cloud computing environments. This architecture allows for the deployment of smaller, more compact draft models directly on edge devices, which rapidly generate candidate tokens. These candidate tokens are then sent to a larger, more powerful verifier model residing in the cloud for parallel validation. The key efficiency gain comes from only requiring the cloud verifier to correct tokens that are rejected by the draft model, rather than generating every token from scratch. This plug-and-play design effectively shifts the majority of the computational burden to the edge, leading to substantial reductions in inference time and cloud costs, all while maintaining the full accuracy of the larger cloud model without any retraining. Experimental results across multiple NLP tasks demonstrate SPADE's ability to reduce cloud model calls by 76% without compromising accuracy.