AdaKP Improves LLM Reasoning in RL with Adaptive Knowledge Selection

Reinforcement learning (RL) is a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs), particularly in challenging domains like competition-level mathematics. However, these tasks often suffer from severe reward sparsity, making effective training difficult. A common mitigation involves injecting "knowledge points" (KPs)—short natural-language hints derived from expert solutions—into the LLM's prompt. Current methods either fix these KPs offline or simply scale the total amount of injected text, missing the opportunity to dynamically select the most informative subset. AdaKP introduces an online adaptive selector that re-chooses the optimal subset of KPs for each problem throughout the RL training process. Its core innovation is an entropy proxy, which efficiently scores a KP by measuring the reduction in next-token entropy it induces, requiring only a single, inexpensive forward pass. This proxy offers a provable bound on its truncation bias, making it a reliable alternative to costly rollout-based estimations. The system integrates several lightweight mechanisms to make this signal practical for online use: a momentum smoother to reduce noise, a retirement-and-revival manager for pruning weak KPs while maintaining exploration, and an adaptive scheduler that prioritizes re-evaluations early in training. AdaKP also includes a pre-flight validation gate to certify the proxy's effectiveness before expensive runs. Implemented as an additive fork to standard RL trainers, AdaKP significantly outperforms static KP selection baselines across eight competition-mathematics benchmarks with negligible added computational cost, highlighting the untapped potential of online, validated KP-subset selection for reasoning-oriented RL.