In-Context Learning in LLMs Explained as Implicit Policy Gradient

Recent empirical observations show that large language models (LLMs) can refine their outputs iteratively by incorporating generated samples and their associated evaluation scores as in-context examples. This research provides a theoretical framework to understand this phenomenon, proposing that score-conditioned In-Context Learning (ICL) fundamentally corresponds to a policy gradient optimization process. The paper offers a constructive proof illustrating how self-attention mechanisms within LLMs can implement a reward-weighted aggregation, drawing a direct analogy to the REINFORCE algorithm under specific configurations of weight matrices. While this correspondence is directional in the hidden-state space and holds precisely under simplifying conditions, the authors empirically quantify its strength, validating the theoretical link. Furthermore, within a simplified hidden-state model, the study derives an exact upper bound on the distribution shift caused by a bounded attention update. This provides a trust-region-like analogy to KL-constrained policy optimization, offering insights into how LLMs adjust their output distributions. Extensive experiments across various LLMs confirm that these models effectively utilize score information to bias their output distributions towards high-scoring exemplars, and that attention weights indeed correlate strongly with example scores.