Runge-Kutta Optimizers Improve Training Loss, Not Generalization.

The application of higher-order Runge-Kutta (RK) integrators to neural network optimizers, inspired by gradient-flow discretizations, has been a topic of interest. This research conducts a stringent, compute-matched evaluation of an RK-Adam variant, a protocol often overlooked in prior studies. The findings reveal that, despite its complexity, the RK variant underperforms plain Adam in terms of training loss when given the same gradient evaluation budget. Further investigation showed that the "adaptivity" of the RK method was largely ineffective; the normalized error remained far below tolerance, and the step size consistently hit its growth cap. When the RK method was properly repaired, it achieved significantly lower training loss (up to 40x better than tuned Adam) in full-batch scenarios. This gain was attributed to an emergent warmup-and-growth schedule from the adaptive control. However, this substantial reduction in training loss did not translate into improved test accuracy or generalization. While gradient averaging in RK provided a small implicit regularization effect, beating lr-matched Adam and AdamW in some cases, cheaper first-order methods like RMSprop and NAdam often matched or surpassed these benefits at a third of the computational cost. The conclusion is that higher-order adaptive integration primarily buys deeper deterministic minimization and minor regularization, without offering advantages over well-tuned, less computationally intensive first-order baselines for generalization.