SAGE Improves RAG Performance with Adaptive Retrieval

Production-grade Retrieval-Augmented Generation (RAG) systems face a critical challenge: balancing strict service level objectives (SLOs) for tail latency and infrastructure costs with the need for high-quality answers. Traditional RAG pipelines often use a fixed number of retrieved passages, which can be inefficient. This leads to over-retrieval for simple queries, wasting resources, and under-serving complex queries, compromising answer quality. To address this, researchers propose SAGE, an SLO-aware adaptive retrieval policy. SAGE intelligently and dynamically selects the optimal number of passages (k) to retrieve for each individual query. It achieves this by using lightweight features derived from initial retrieval steps, such as score distributions and lexical signals. The policy is trained offline using imitation learning, mimicking an oracle that understands the best latency-quality trade-offs. Crucially, SAGE adds minimal overhead during inference and requires no additional LLM calls. Evaluations on datasets like Natural Questions demonstrated its effectiveness: under a 5-second P95 latency SLO, SAGE achieved 95% compliance compared to 30% for the best static baseline. It also reduced P95 latency by 36% and retrieval cost by 51%, with only a minor 2 percentage point loss in Exact Match accuracy. A single SAGE policy trained on one dataset proved capable of generalizing across multiple other datasets and various LLM families, consistently delivering substantial SLO improvements without degrading quality.