SAGE Improves LLM Unlearning by Preserving Retained Knowledge

Machine unlearning in Large Language Models (LLMs) aims to remove specific, undesirable knowledge or behaviors while ensuring that the model's general capabilities and retained knowledge are not compromised. A common challenge is the inherent trade-off: methods designed to forget often inadvertently degrade the model's ability to retain other important information. This paper introduces SAGE, a post-hoc sanitization approach that addresses this trade-off without requiring a rerun of the original unlearning process. SAGE leverages the "retention activation bias" to quantify the damage an unlearning method inflicts on retained capabilities. SAGE collects real module inputs from a small "retain proxy" and extracts their dominant activation geometry. It then solves a source-anchored optimization objective in closed form, which suppresses update components aligned with high-energy retained directions while preserving the forgetting mechanism of the original unlearning method. Across various unlearning methods, model scales, and benchmarks, SAGE consistently improves the retain-forget trade-off, highlighting the potential of post-hoc vector sanitization.