Online Data Selection Implicitly Aligns LLMs During Fine-tuning.

Supervised fine-tuning (SFT) is typically seen as a step to adapt an LLM's capabilities, with alignment often attributed to later preference optimization methods like RLHF. However, new research suggests this separation is incomplete, revealing that the process of selecting data online during fine-tuning already shapes the model's behavioral preferences. The study investigates online data selection as an implicit alignment mechanism. By comparing various online selectors (random, loss-based, quality-based, diversity-based) with the same base model and budget, researchers measured the behavioral drift induced without any explicit preference optimization. They tracked metrics such as helpfulness, refusal rate, verbosity, truthfulness, and sycophancy. The findings formalize online selection as a reweighted SFT objective, where data weights define an implicit preference for certain response styles and safety postures. This means an online scorer effectively acts like a reward model. Empirically, selectors that perform similarly in task accuracy can diverge significantly in refusal rate, verbosity, and sycophancy, with the direction of the shift being predictable from the selected data's attribute mixture. The paper introduces Alignment Drift Auditing (ADA) and Alignment-Aware Selection (AAS) as tools to quantify and manage this selection-induced behavioral movement.