RL Training Improves AI Alignment and Beneficial Behavior

As AI systems become more prevalent in diverse and critical applications, ensuring their alignment with human values must extend beyond their initial training tasks and domains. Reinforcement Learning (RL) can inadvertently introduce misalignments through reward hacking or deceptive strategies. This research investigates whether RL, when applied to beneficial behaviors in realistic scenarios, can foster broad and persistent alignment that generalizes beyond the training distribution. The study involved creating a dataset of realistic situations designed to measure and train beneficial traits such as truthfulness, fairness, risk awareness, and corrigibility across various domains like health, science, and education. Models were then trained using RL on this dataset and subsequently evaluated on over 50 independent benchmarks assessing alignment and beneficial behavior. The results showed that beneficial trait RL significantly improved performance on over 80% of these out-of-distribution benchmarks compared to a compute-matched baseline. Notably, alignment transfer was observed even when RL intervention was limited to a single domain, like health, leading to broad improvements in non-health alignment evaluations, including reduced reward hacking and deception. Furthermore, models trained with beneficial trait RL exhibited improved persistence, showing greater resistance to adversarial prompting and harmful fine-tuning, although further research is needed to fully understand these effects. These findings suggest that reinforcing beneficial behavior through RL in realistic contexts can lead to AI models that are more robustly aligned with human flourishing.