Adaptive Optimizer Selection Boosts Deep Learning Performance

In deep neural network training, the choice of optimizer is a critical hyperparameter, typically fixed before training begins. Treating it as such often requires multiple complete training runs to find the best one, a computationally expensive process that discards all but the winning model. This research proposes Repeated Optimizer Resampling (ROR), an innovative approach that allows for adaptive optimizer selection within a single, evolving training run. The method works by periodically (e.g., every 'b' epochs) having each candidate optimizer "scout" from the current model weights for a short duration (e.g., 's' epochs). The optimizer that performs best during this scouting phase is then used for the remainder of the 'b-s' epochs. If this completed segment improves the validation objective, it becomes the new incumbent. This dynamic design allows the preferred optimizer to change as training progresses, adapting to different phases of the optimization landscape. Experiments on MNIST, Fashion-MNIST, and two motor insurance claim-count models showed that ROR, particularly a one-epoch scouting variant, achieved performance close to the exhaustively identified best fixed optimizer while using only 24% to 35% of the aggregate training time.