New Framework Predicts Post-SFT Model Mechanisms for Better Tuning

This paper introduces a novel framework designed to overcome a fundamental limitation in mechanistic interpretability: its retrospective nature. Traditionally, interpreting pre-fine-tuned models to guide subsequent Supervised Fine-Tuning (SFT) can lead to misleading conclusions because the critical mechanisms often shift significantly after tuning for a new task. The proposed "forward-looking localization" framework aims to predict these post-SFT mechanisms using only the pre-SFT model parameters and the target dataset. The theoretical foundation models SFT as a continuous parameter evolution, leveraging Taylor expansion to connect the post-tuning mechanistic objective with the pre-SFT model's dynamic gradients. Practically, the framework employs dual-granularity localization pipelines, operating at both neuron and component levels. Extensive experiments demonstrate that this approach not only provides superior guidance for SFT but also exhibits robustness and scalability across increasing model sizes, pioneering a predictive approach to uniting interpretability with targeted optimization.