When Neural Networks Improve Local Approximation in Optimization

The effectiveness of neural surrogates in derivative-free optimization has often appeared contradictory, with similar models yielding vastly different results across solvers. This study resolves these inconsistencies by pinpointing three crucial factors that determine when a learned local model provides a benefit, rather than simply relying on training curve accuracy. The first factor is the surrogate's role: surrogates that propose candidate solutions for the true objective to approve are beneficial, whereas those that replace a gradient upon which the solver depends can be detrimental. The radius of reliability is the second factor; a model fitted to an optimization path is only dependable within a bounded neighborhood, with its error neither vanishing as the neighborhood shrinks nor surviving its expansion. Finally, the room for improvement is critical: a surrogate can only accelerate progress that the base optimization method is still capable of making. Through experiments on 117 benchmark instances, the research demonstrates that safeguarded assistance from surrogates increases solved instances to high accuracy from 67 to 84, while gradient replacement lowers it to 65. Removing the gradient term from training loss drastically cuts surrogate acceptance, and the study shows no noise threshold, only a base method that stops early.