Akashic Improves LLM Inference Efficiency with Novel Memory System.

Large Language Model (LLM) agent systems often struggle with the increasing context length required for multi-turn interactions and complex workflows. Constantly replaying the entire conversation history leads to high prefill costs, context limit issues, and can degrade output quality by burying relevant information. This new research introduces Akashic, a memory system designed to address these challenges. Akashic employs a technique called MemAttention, which structures context into manageable, bounded chunks. This method allows the system to model semantic connections across these chunks, ensuring that crucial cross-chunk evidence is retained without the need to repeatedly process the full historical context. Furthermore, Akashic incorporates hardware-software co-design for memory placement, strategically co-locating frequently retrieved chunks to minimize retrieval fragmentation and I/O overhead. Evaluations across various workloads and model sizes demonstrate Akashic's effectiveness. It has shown improvements in task accuracy by up to 10.2 points, increased throughput by up to 1.21x, and boosted sustainable request rates by up to 1.88x compared to existing memory baselines. This indicates a significant step forward in making long-context LLM applications more efficient and performant.