New Protocol Improves LLM Unlearning and Knowledge Retention

Machine unlearning in large language models (LLMs) is crucial for privacy and safety, aiming to remove specific knowledge while preserving all other capabilities. However, current evaluation methods are unreliable, often failing to detect if forgotten knowledge resurfaces through paraphrased queries (under-forgetting) or if unrelated knowledge is inadvertently lost (over-forgetting). This problem is framed as an asymmetric generalization challenge. Effective forget evaluation requires testing diverse query formulations, while retain evaluation must probe a vast, implicitly defined set of unrelated facts. Existing datasets lack the fine-grained annotation needed to define this forget-retain boundary accurately. To address these shortcomings, a new evaluation protocol and training corpus called SUITE has been developed. SUITE captures the intricate forget-retain structure for real-world factual domains. Training methods on SUITE significantly improve unlearning outcomes, highlighting the importance of data quality. Building on these insights, the researchers also introduced JensUn++, an unlearning algorithm that demonstrates superior forget-retain utility across multiple LLMs in both sequential and joint unlearning scenarios.