New Benchmark Tests LLM Unlearning Robustness

Ensuring that large language models (LLMs) can effectively "unlearn" sensitive or outdated information is crucial for privacy and compliance. Current benchmarks for machine unlearning primarily focus on single-hop questions or a limited set of multi-hop queries, which may not fully capture the complexities of knowledge removal. This research highlights two significant challenges: the potential for knowledge leakage through diverse multi-hop reasoning paths and the fragility of unlearning, where removed knowledge can be partially recovered through lightweight post-unlearning adaptation attacks. To address these limitations, a novel benchmark called "Leak-Resistant Unlearning" is introduced. This benchmark is designed to rigorously assess the robustness of LLM knowledge removal across a wider array of reasoning paths and against various recovery attacks. Experiments conducted on multiple models and unlearning methods using curated datasets demonstrate that existing techniques are indeed vulnerable, underscoring the need for more robust unlearning strategies that can withstand sophisticated attempts at knowledge recovery.