SAGE Improves LLM Unlearning by Preserving Retained Knowledge

Large Language Model (LLM) unlearning is a critical area focused on removing specific, unwanted information or behaviors from models while ensuring that their core, desired knowledge and abilities are preserved. A common challenge in current unlearning techniques is the inherent trade-off: improving unlearning often comes at the cost of degrading the model's retained capabilities. Researchers have identified that the bias in retention activation can serve as a quantifiable measure of the damage an unlearning method inflicts on a model's retained knowledge, independent of the specific unlearning process used. This insight enables a novel post-hoc approach to restore retention performance. The proposed method, SAGE (Spectral Activation-GEometry Sanitization), is a source-agnostic correction applied to the final unlearning update vector. It works by analyzing real module inputs from a small proxy, identifying dominant activation geometries, and then optimizing to suppress update components that align with high-energy retained directions, all while maintaining the original unlearning method's forgetting mechanism. SAGE has been shown to consistently alleviate the retain-forget trade-off across various unlearning methods, model scales, and benchmarks.