Hypernetwork Enables More Efficient Differentially Private Learning

Differentially private (DP) training of neural networks often faces challenges due to the substantial noise required by gradient-based methods like DP-SGD. These methods repeatedly inject high-dimensional noise into the parameter space throughout the training process, which can degrade model utility. This research introduces a novel DP learning framework designed to circumvent this iterative noise injection. The proposed approach employs a hypernetwork, trained on public datasets, to generate the target model parameters from a private dataset. Each private example is first embedded into a low-dimensional representation, which is then aggregated and perturbed to create a DP dataset embedding. The hypernetwork then uses this single, low-dimensional noisy embedding to produce the final model parameters. By injecting privacy noise only once into a low-dimensional representation, this method significantly reduces the negative impact of noise, leading to higher utility compared to DP-SGD, as demonstrated theoretically and through applications like LoRA fine-tuning of diffusion models.