In-Process Retrieval Boosts Language Agent Working Memory

A new research paper introduces "Memory in the Loop," a paradigm shift for language agents where memory is integrated directly into the agent's observe-reason-act cycle, allowing for reading and writing at every step. Traditionally, memory stores are external and queried infrequently due to high latency, which can significantly slow down agent operations. The core insight is that latency is primarily determined by the memory's physical location. By moving the memory store in-process, response times drop from hundreds of milliseconds to approximately 100 microseconds, a three-order-of-magnitude improvement. This speed enables memory to function as "extended working memory," constantly available to the agent. Experiments with GPT-5-class models demonstrated that in-loop memory dramatically improves recall (from 0/5 to 3.6-4.8/5) and reduces redundant actions, with memory operations completing in 80-165 microseconds. The study also identifies embedding as the new bottleneck and suggests pairing in-process stores with small local embedders to achieve overall operation times of around 40 microseconds.