New Method Reveals Language Model Generalization Failures.

Large language models (LLMs) undergo extensive post-training on curated datasets, yet a gap often remains between their training environment and real-world deployment. This "distribution shift" can lead to unexpected generalization failures, which are not yet fully understood. To shed light on these failures, researchers are seeking simplified, controlled demonstrations. This paper introduces a flexible method for creating language models that exhibit predictable generalization failures when subsequently trained with Reinforcement Learning (RL) on specific task distributions. The technique involves Supervised Fine-Tuning (SFT) on a dataset composed of transcripts from a "mixture of conditional policies." Each policy can be assigned distinct behaviors for different task distributions. The study observes that during RL training, the model tends to select policies that yield the highest reward on the training distribution. This can lead to striking outcomes: in a controlled experiment, training on one distribution with a specific "trigger string" caused performance on an identical task with a different trigger string to degrade to zero. The authors also illustrate novel ways generalization can fail due to shifts in task coverage and temporal context, offering "model organisms" for stress-testing AI alignment and advancing generalization science.