Forward-Forward Learning's "Goodness" Explained by Likelihood Ratios

The Forward-Forward (FF) algorithm, a method for training neural network layers locally, relies on a "goodness" score derived from the sum of squared activations. This score is designed to be high for real inputs and low for contrastive ones, with activations normalized between layers. Previously, these design choices were largely considered heuristics. This new research offers a rigorous theoretical explanation, demonstrating that the squared goodness metric is, in fact, the sufficient statistic of a likelihood-ratio test between two distinct data populations. The study extends this understanding to various data distributions, showing how anisotropic populations lead to a Mahalanobis goodness and heavy-tailed populations benefit from a saturating statistic. Furthermore, the paper clarifies that inter-layer normalization is crucial for preserving per-coordinate energy while removing length, thereby preventing a phenomenon known as depth collapse.