Ingest-Time Compilation Boosts RAG Performance, Reduces Cost

Current Retrieval-Augmented Generation (RAG) systems often suffer from a fundamental inefficiency: they repeatedly re-interpret raw corpus text with a language model during each query, discarding that computational effort afterward. This "full-table scan" approach leads to escalating inference costs, even as per-token prices decrease, because context volume grows faster than price reductions. The paper proposes a solution inspired by database indexing: "ingest-time semantic compilation" (ISC). This paradigm involves performing the expensive semantic processing once, at write time, by compiling the corpus's meaning into a queryable, incrementally maintained substrate. This substrate consists of coupled layers, including embeddings and atomic claims whose provenance is validated during compilation, treating it as a first-class database object. Two proofs of concept support ISC's efficacy. Firstly, substrate upkeep scales with change, not corpus size, with incremental updates being 33.7 times cheaper than full reconstruction while maintaining precision. Secondly, using compiled claims as the retrieval payload significantly outperformed traditional chunk configurations across 500 broadcast-interview transcripts, achieving 85.2% correctness with fewer reader tokens (2.2k vs. 16.3k). The only comparable baseline was a contextualized-chunk pipeline with hybrid retrieval and reranking, which the authors argue achieves parity precisely because it implicitly performs a form of compilation. The research concludes by outlining a systems agenda for ISC, including compilation planners and read planning.