SFT Lessons Transfer Across AI Alignment, Model Organisms, and Toy Models.

This paper investigates the transferability of supervised fine-tuning (SFT) techniques and insights across seemingly distinct domains within AI research: alignment training, model organisms, and toy models. Despite their differences, these areas often employ SFT to achieve similar underlying objectives, prompting the question of whether lessons from one can benefit the others. The authors demonstrate this transferability through three specific experiments. Firstly, a lesson from alignment training regarding behavior generalization, specifically that training on the *reason* for a behavior (like "Teaching Claude Why") leads to better generalization than training on examples alone, was successfully applied to toy models. Secondly, a finding from model organisms about capability preservation during SFT, where training on "off-model" outputs can degrade capabilities, was tested in an alignment setting. It was found that incorporating benign "on-model" data can mitigate this damage while still embedding the desired behavior. Finally, a lesson on robustness from model organisms was ported to the same alignment context, revealing that subsequent benign SFT can erase aligned behaviors even if capabilities are preserved, indicating that capability preservation alone does not ensure long-term robustness. This work highlights the value of cross-domain knowledge sharing in SFT research, suggesting a more integrated approach to developing and refining AI training techniques.