PAC-Bayes Bounds Improve Model Generalization with Geometric Priors

Overparameterized machine learning models often have multiple parameter settings that result in the same predictive behavior. This redundancy, known as continuous parameter symmetries, can complicate the analysis of model generalization. This paper proposes a novel PAC-Bayesian approach that addresses this by performing analysis on the "quotient predictor space," effectively treating equivalent parameterizations as a single entity. The core innovation involves constructing a canonical prior distribution that inherently reflects the model's implicit biases, derived from the geometric volume of these equivalent parameterizations. This "geometry-induced implicit-bias prior" offers a more accurate representation of the model's true learning tendencies compared to traditional neutral priors. Empirical tests in Fourier regression and Query-Key attention models demonstrate that this method significantly reduces the Kullback-Leibler (KL) divergence in the quotient space and tightens PAC-Bayes generalization certificates, even without explicit regularizers. This suggests a more robust way to understand and bound the performance of complex models.