Adaptive Compression Boosts Edge RAG Performance and Efficiency

Retrieval-Augmented Generation (RAG) systems enhance language model responses by incorporating external information, but this comes with a cost: longer prompts increase computational overhead on edge devices, impacting latency and energy consumption. While context compression can mitigate this by pruning retrieved text, current methods often use fixed compression rates, ignoring dynamic workload variations and the live state of the edge device. Researchers introduce a vision for telemetry-informed adaptive compression for edge-based RAG. This approach dynamically adjusts the compression rate at runtime, guided by workload features and real-time edge device telemetry. Experiments on an NVIDIA Jetson AGX Thor, using Llama and Qwen generators with LLMLingua-2 compression, demonstrated the benefits. The findings show that generation is the dominant factor in RAG's energy and latency budget for larger models. Crucially, intermediate compression rates were found to be optimal, reducing GPU energy by up to 53.2% and SoC energy by up to 48.2% with negligible impact on quality. Overly aggressive compression can degrade quality, while mild compression misses efficiency opportunities. This research advocates for runtime policies that intelligently manage compression to optimize RAG performance on edge devices.