New Auditor Evaluates AI Unlearning Algorithm Effectiveness

A new practical auditor has been developed to rigorously assess the effectiveness of machine unlearning algorithms. This tool addresses the challenge of verifying whether these algorithms truly eliminate the influence of specific training data from a model. The auditor operates by computing data-dependent lower bounds on the unlearning parameter using membership inference attacks. Applying this auditor to various unlearning algorithms revealed a clear distinction in their performance. Algorithms with strong theoretical guarantees, such as model clipping and rewind-to-delete, achieved very small unlearning parameter bounds, thereby validating their claims. In contrast, empirical methods like Hessian-based unlearning and fine-tuning on the retain set exhibited large bounds, indicating they failed to adequately unlearn data. This auditor provides a robust, hypothesis-testing framework for empirically falsifying unlearning claims, validated on datasets like CIFAR-100 and Shakespeare text. It offers a crucial tool for ensuring data privacy and compliance in AI systems.