New Architecture Improves Verbal Reinforcement Learning with Insight Governance

Training-free verbal reinforcement learning allows large language model (LLM) agents to learn from real-world feedback, such as task outcomes or market returns, by extracting verbal rules and using them as context to update behavior without altering model parameters. However, in dynamic, non-stationary environments, these agents face a critical "retention-forgetting dilemma": retaining outdated insights can lead to negative transfer, while discarding them can result in catastrophic forgetting when similar conditions reappear. The researchers identify four key requirements to navigate this dilemma: outcome-driven evaluation, persistent structured evidence, a non-monotonic knowledge lifecycle, and compositional governance. They observe that existing methods often focus heavily on extracting experience but underinvest in the crucial aspect of insight governance. To bridge this gap, a new three-layer architecture is proposed, comprising rules, evidence, and skills, interconnected by a feedback-driven curation loop. "Rules" capture distilled experience from world outcomes, "evidence logs" track each rule's reliability over time, and "skills" govern which rules to apply, how to resolve conflicts, and when to abstain. Using financial forecasting as a case study—an environment known for its abundant, noisy, and non-stationary feedback—the study demonstrates that the same accumulated experience can either degrade performance below a zero-shot baseline or dramatically enhance accuracy and risk-adjusted returns, depending on the presence and effectiveness of this curation loop.