New Method Enhances LLM Self-Improvement with Procedural Memory

Current reinforcement learning methods for language models, like RLVR and SDPO, often discard rich procedural information from rollouts, focusing only on episode-level signals. This new research introduces Procedural Memory Distillation (PMD), a technique that enables language models to retain and reuse cross-episode information. PMD converts these signals into a structured procedural memory, which is then distilled directly into the model's weights during training. The memory is organized at three levels: raw trajectories, self-reflected strategies, and recurring behavioral patterns, all extracted online from the model's own interactions. A memory-conditioned self-teacher uses this accumulated experience to supervise the student model, allowing it to progressively internalize procedural knowledge. This "co-evolution" design, where the policy updates memory and memory shapes policy updates, is central to its effectiveness. Empirical results show PMD significantly outperforms SDPO on benchmarks like SCIKNOWEVAL and LIVECODEBENCH, with gains of 3.8-5.5% and 7.9-13.6% respectively. The co-evolution aspect is crucial, as freezing either the memory or the policy leads to substantial performance drops. This approach yields a memory-free model at inference, as the knowledge is absorbed into the policy itself.