Revisiting Volume Hypothesis in Deep Learning Generalization.

The remarkable generalization ability of over-parameterized deep neural networks, despite having more parameters than training data, is often attributed to the implicit bias of stochastic gradient descent (SGD). An alternative explanation, the "volume hypothesis," suggests that regions of the loss landscape leading to good generalization are simply much larger in weight space, making them more likely targets for optimization algorithms. Previous experimental studies on this hypothesis have yielded seemingly contradictory results, with some supporting it and others not. This paper attempts to reconcile these findings by exploring an intermediate dataset size regime. Using the Replica Exchange Wang-Landau algorithm, the researchers estimate the joint density of states over training and test accuracies in binary networks. Their observations indicate that the generalization advantage of gradient-based learning over random sampling diminishes as the training data size increases, offering a potential resolution to the observed paradox.