LLM-as-a-Tutor Adapts Prompts for Better RL Training.

In reinforcement learning (RL) tasks where human verification is difficult, large language models (LLMs) are often used as judges to provide reward signals. However, the training prompts typically remain static, leading to a mismatch between prompt difficulty and the agent's evolving skill level. This can result in the LLM judge failing to provide useful feedback when prompts are either too easy or too hard. To overcome this, a new framework called "LLM-as-a-Tutor" proposes an LLM that serves a dual role: an examiner and a generator. As an examiner, it compares agent rollouts to identify prompts that no longer challenge the policy. As a generator, it appends specific constraints to these prompts, incrementally increasing their difficulty. This "append-only" design ensures that the training signal continuously calibrates with the policy's capability without needing external difficulty schedules. Experiments across various instruction-following benchmarks show that this method consistently outperforms existing approaches that only adapt rubrics or rewrite prompts, highlighting the importance of dynamic prompt adaptation in non-verifiable RL.