PAC-Bayes Theory Decomposed for Predictive Behavior Complexity

This research delves into PAC-Bayes theory, a framework for providing generalization guarantees in machine learning. The core insight is that traditional PAC-Bayes measures of complexity often conflate uncertainty about a model's actual predictions with the variability among different internal configurations that produce the same predictive outcome, especially in over-parameterized systems. The authors introduce a novel structural decomposition of the classical PAC-Bayes Kullback-Leibler (KL) divergence. This decomposition separates the complexity into two components: one related to selecting predictive behaviors and another concerning the specific internal realizations that implement those behaviors. They define "Z-information" as a metric that quantifies the difference between the overall KL divergence and the complexity solely attributable to predictive behavior uncertainty. Furthermore, the paper demonstrates that the behavior-selection component can be characterized variationally, and that concepts like symmetry and fiber geometry naturally emerge from this behavior-map structure. These findings aim to refine how model complexity is understood and measured in the context of generalization.