Diff Mining Reveals Finetuning Objectives in Language Models

Finetuning is a standard practice for adapting large language models (LLMs) to specific tasks or behaviors. However, understanding precisely what new behaviors emerge during this process, especially concerning the potential for unwanted or biased outcomes, remains a significant challenge. Many existing methods for analyzing model changes require access to internal model structures, limiting their applicability to large, proprietary models. A new framework, dubbed Diff Mining, offers a simple yet powerful solution. It works by comparing the output logits (raw prediction scores) of a finetuned model against its base model on a reference corpus. This comparison highlights tokens that are significantly amplified or suppressed by the finetuning, effectively creating a 'fingerprint' of the training objectives. Crucially, this analysis can be performed even on text outside the finetuning domain, and it only requires access to the model's outputs, making it scalable to very large models. Diff Mining operates in two stages: first, extracting per-context logit differences, and second, aggregating these signals to form an interpretable set of tokens. The aggregation can use simple frequency methods or more advanced techniques like Non-negative Matrix Factorization to disentangle multiple finetuning objectives. Empirical results show Diff Mining outperforms state-of-the-art model diffing methods in identifying relevant tokens for domain detection and effectively uncovers injected biases, proving its utility as an auditing tool for finetuned LLMs.