Predicting Neural Network Training with Low-Dimensional Reparameterizations

Training or fine-tuning neural networks can sometimes be achieved using random low-dimensional reparameterizations, where a small latent vector is mapped to a full parameter update via a fixed random map. A key practical question arises: how large must this latent search space be to successfully reach a low-loss region? This research addresses this by first expressing the known accessibility transition in an equivalent conic form. The main theoretical contribution is an orientation-resolved quadratic master formula. This formula predicts the random-slice residual by considering both the curvature spectrum and the displacement profile from the reference to the solution. This leads to a self-consistent isotropic-orientation predictor and, in a simplified form, recovers earlier Gaussian-width quadratic bounds. Building on this analysis, the researchers introduce Random Mapping Networks (RaMaN). RaMaN instantiates the predicted latent dimension using structured Hadamard or seed-regenerated Gaussian maps, which significantly reduces the memory footprint for optimizer state from O(P) to O(d), avoiding the O(dP) storage of dense random maps. The framework also includes matrix-free curvature approximations and sweep-free dimension selection. Experiments across quadratic and neural-curvature settings show the orientation-resolved predictor accurately tracks transition locations, outperforming orientation-agnostic approximations. End-to-end experiments further demonstrate sharp, protocol-dependent training transitions across image and language models.