Kalman-Guided Prompt Selection Boosts RL Finetuning for LLMs

This paper addresses a critical challenge in finetuning large language models (LLMs) using reinforcement learning (RL): the effective selection of prompts. The success of RL finetuning heavily relies on presenting prompts of appropriate difficulty, which constantly changes as the policy evolves during training. Existing online methods struggle with this, either being accurate but computationally expensive or efficient but assuming static difficulty. The proposed solution, Kalman-Guided Prompt Selection (KGPS), reframes prompt selection as a dynamic state estimation problem. KGPS models each prompt's latent success rate using a linear-Gaussian state-space model, where uncertainty increases with policy updates. A Kalman filter then maintains a calibrated posterior over prompt difficulty, enabling the selection of intermediate-difficulty prompts while naturally revisiting uncertain ones. KGPS is adaptive to policy drift and requires no additional rollouts beyond standard policy training. Extensive experiments across various reasoning benchmarks (mathematics, planning, geometry) and RL algorithms demonstrate that KGPS consistently outperforms strong baselines in both final accuracy and rollout efficiency. For instance, it achieved an 83% reduction in rollouts for DeepSeek-R1-Distill-7B while improving average performance.