SPSD Compresses LLM Prompts at Edge, Reduces Cloud Energy Costs

The "prefill" stage of Large Language Model (LLM) inference is a major contributor to the energy consumption of cloud-scale AI. Many user prompts, especially in conversational or customer support contexts, contain "social scaffolding" – polite phrases, apologies, and repetitions – that are important for human interaction but carry little semantic information for machine reasoning. This disparity is termed the "Social-Semantic Gap." To address this, researchers propose SPSD (Sentiment Preserving Semantic Distillation), an edge-based pipeline. This system utilizes a 4-bit quantized Small Language Model (SLM) to compress user prompts directly on the user's device before they are transmitted to a larger, cloud-deployed LLM. This process aims to remove the non-essential social scaffolding while retaining the core semantic and emotional content. Evaluations using Gemma-2-2B-Instruct as the SLM and Llama-3.1-8B-Instruct as the cloud model showed significant results. The system achieved a mean saving of 99.9 input tokens per distilled call, with all tested distilled calls yielding positive savings. Response quality, assessed by a blind LLM-as-judge method, was found to be non-inferior to raw prompts within a specified margin, with a good balance of ties, distilled wins, and raw wins. Estimated per-call net energy savings range from 70-270 uWh. For safety-critical domains, a rule-based gate ensures conservative routing to passthrough, highlighting a practical approach to reducing cloud LLM costs while maintaining quality.