MemOPD Improves Long-Horizon Agent Learning with Memory Alignment

Training AI agents for long-horizon tasks often leads to performance degradation and instability as the agent accumulates vast amounts of context over time. While compact memory mechanisms can compress and rewrite historical data, learning what information to retain is challenging, especially with sparse task rewards. On-policy distillation (OPD) offers a solution by providing dense supervision from a teacher model during student rollouts.However, a critical issue arises in standard OPD when memory compression is involved: the teacher model might evaluate an action under a state that the student never actually experienced during its rollout, due to context rewriting. This "state misalignment" invalidates the teacher's supervision. To overcome this, researchers developed Memory-Aligned On-Policy Distillation (MemOPD).MemOPD meticulously records the inputs and outputs of each model invocation, then reconstructs the original token positions and causal visibility. This allows for efficient and accurate teacher scoring under the exact state the student encountered. The teacher provides full-vocabulary supervision for sampled actions, while Proximal Policy Optimization (PPO) maintains the overall task objective. Experiments show MemOPD significantly improves performance, particularly on F1 scores, and offers substantial speedups in actor computation during training.