Lightweight Chunk Selection Boosts Mobile RAG Performance

Deploying Retrieval-Augmented Generation (RAG) models on mobile and edge devices faces significant challenges due to the computational and memory demands of retrieved context. A common approach to reduce costs is to use only one retrieved chunk, but the highest-ranked chunk isn't always the most relevant or evidential. Existing methods for improving context quality often require additional, costly LLMs or compressors, which are impractical for strict mobile budgets. This paper proposes a lightweight chunk selection method that frames the problem as evidence alignment. It integrates three key feature sources: the LLM's internal query intent, expert signals derived from Mixture-of-Experts (MoE) routing, and the embeddings of retrieved chunks. These features are mapped to an "evidence prototype" using a compact multilayer perceptron, and the chunk most aligned with this prototype is selected. The method consistently improves rank-1 evidence selection by an average of 2.5% over mobile-applicable baselines. This demonstrates that leveraging LLM-side query representations and MoE routing, combined with retrieval-side candidate embeddings, offers an effective and parameter-efficient strategy for enhancing RAG performance on mobile devices.