New DP-NGD Method Boosts Private AI Training Efficiency

Traditional differentially private optimization methods often struggle with efficiency, leading to either premature training stops or excessive noise injection, both of which compromise model accuracy. This inefficiency stems from their reliance on local gradients, ignoring the loss function's curvature. Natural Gradient Descent (NGD) offers a solution by using curvature information to guide updates more effectively, but its direct integration with differential privacy (DP) presents significant technical hurdles. A new framework, DP-NGD, has been developed to overcome these challenges. It innovatively decouples curvature estimation from private data, aligns isotropic DP constraints with NGD's anisotropic scaling, and dynamically clamps curvature to ensure training stability. This systematic approach allows DP-NGD to leverage the benefits of NGD within a privacy-preserving context. Extensive experiments demonstrate that DP-NGD achieves state-of-the-art accuracy, surpassing first-order DP baselines. It also offers up to a 10x speedup in convergence under the same privacy budget, effectively breaking the long-standing privacy-utility trade-off bottleneck in AI training.