Introspection Adapters Detect Fine-Tuning Side-Effect Misalignments

Fine-tuning large language models (LLMs) allows them to acquire new capabilities but can inadvertently degrade existing alignment properties, leading to "side-effect misalignments." While previous work used introspection adapters (IAs) to explain explicitly implanted behaviors, the challenge of detecting unintended alignment shifts from fine-tuning on unrelated tasks remained. This paper bridges that gap by formulating the problem of side-effect introspection. The study constructs a new dataset specifically for this setting, enabling the investigation of how fine-tuning for one task can unintentionally impact other alignment aspects like safety or ethical behavior. To improve the detection of these subtle internal model changes, the authors propose the Delta-Aware Introspection Adapter (DAIA). DAIA is designed to explicitly process both the base model's activations and the differences in activations induced by the fine-tuning process. Empirical evaluations demonstrate that DAIA consistently outperforms existing introspection adapters, showing improved generalization to unseen fine-tuned models and various safety categories. This advancement provides a crucial tool for understanding and mitigating unintended consequences of LLM fine-tuning.