Semantic Compression Trees Offer Hierarchical Knowledge Retrieval.

Retrieval-augmented generation (RAG) typically relies on flat, fixed-granularity indexes, where documents are broken into uniform chunks and retrieved based on similarity. This approach often disregards the inherent hierarchical structure of source documents. Researchers propose Semantic Compression Trees (SCT) as a new hierarchical indexing method. In SCT, each node stores only its "semantic residual," representing the unique information it adds beyond its parent node. Retrieval then proceeds by progressively descending the tree, making the per-query cost dependent on tree depth rather than the entire collection size. Evaluations on the QASPER dataset yielded mixed results. When the relevant document was provided, SCT with an extractive compressor matched dense retrieval in answer quality while using 30% fewer context tokens and requiring no LLM calls for index building. Residual storage also proved more efficient than storing full summaries. Furthermore, scaling the collection fifty-fold increased flat retrieval's scoring work by nearly 49x, compared to only 6.4x for SCT. However, the progressive descent routing mechanism itself did not perform well. When the system had to select the document, descent was substantially worse than flat retrieval, primarily because the initial document choice is made from the root residual, the most compressed node. The study concludes that while the residual representation is valuable, the top-down routing strategy needs further improvement.