Bridging Privacy Gap in DP-SGD with Enhanced Empirical Defenses.

Differential privacy (DP) provides theoretical guarantees for data privacy in machine learning, but achieving strong utility-privacy trade-offs remains challenging. Recent work in privacy auditing has shown that the true privacy of algorithms like DP-SGD (the de facto private training method) might be lower than theoretical bounds suggest, with empirical attacks nearly matching these bounds. This indicates a "provable-auditable privacy gap." This research proposes a novel perspective: optimizing for the empirical privacy lower bound as a concrete metric. It introduces a lightweight defense framework that can be generically applied to augment existing optimization methods in the ML pipeline. This framework is shown to significantly improve empirical privacy on standard benchmarks. Crucially, unlike other defenses against membership inference attacks, this new framework achieves its empirical privacy gains without any additional theoretical privacy cost when augmenting DP-SGD. The defense's flexibility is demonstrated through evaluations against a wide range of audit constructions, models, and datasets, suggesting a practical path to enhancing privacy in real-world ML deployments.