Fisher8 Stabilizes Neural Networks for Better Uncertainty Prediction

Training neural networks to simultaneously predict both the mean and the uncertainty (variance) of observations, known as heteroscedastic regression, often faces stability challenges. Various independent efforts have attempted to stabilize these models, suggesting a common underlying problem related to how gradient updates interact with the loss landscape. Researchers propose Fisher8, a novel output-layer gradient correction technique. This method reorients and rescales gradient updates by leveraging Fisher geometry, rather than the standard Euclidean geometry, to better align with the local curvature of the loss function. Unlike previous stabilization methods, Fisher8 avoids introducing additional data-dependent hyperparameters. The study demonstrates that Fisher8 achieves superior likelihood-error trade-offs, produces more accurately calibrated uncertainty estimates, and facilitates the learning of richer, uncertainty-aware feature spaces across diverse multidimensional regression and representation-learning tasks. This advancement offers a more robust and principled way to train models that provide reliable uncertainty quantification.