In-Context Learning Explores Intrinsic Curiosity for Data Selection

Effective machine learning relies not only on sophisticated data modeling but also on intelligent data selection. The concept of "intrinsic curiosity" aims to automate data selection by rewarding an agent for its "learning progress," which measures how much new observations improve a world model's predictive ability. Traditionally, calculating these rewards involves computationally expensive gradient descent updates within each trajectory, making it impractical for large-scale applications. This study explores whether the emergent in-context learning (ICL) abilities of sequence models can overcome this computational bottleneck by acting as immediate, update-free world models. The goal is to train an exploration policy that maximizes learning progress using only an in-context learner's prediction errors and counterfactual context manipulations. The findings indicate a theoretical limitation: in general Markov decision processes, unbiased estimation of true learning progress using ICL-derived rewards is not possible due to nuisance terms. However, a positive result is shown for a broad subclass of non-temporal settings, such as active learning and Bayesian Experimental Design, where ICL-derived rewards can successfully bound and asymptotically converge to true learning progress. Experimental validation in continuous and symbolic environments supports these theoretical claims, demonstrating that this ICL-driven framework can train curious data-collection policies that explore optimally in specific contexts.