DREG Regularization Boosts Neural Network Accuracy and Robustness

New research presents an extensive empirical analysis of Derivative Regularization (DREG), a penalty mechanism for neural networks. The study, involving 960 experiments across various configurations, aimed to pinpoint when, where, and why DREG is effective. Key findings indicate that DREG consistently achieves higher overall accuracy compared to other regularizers, including unregularized baselines and Weight Decay, and shows strong performance in noisy environments. Notably, DREG excels when used with GELU activations, which are standard in modern transformer architectures, and performs particularly well on complex vision and natural language processing tasks. Its benefits are most pronounced in scenarios with limited data, suggesting it provides a valuable geometric inductive bias that compensates for data scarcity. The research highlights DREG's utility as a "plug-and-play" regularizer, requiring minimal hyperparameter tuning, by focusing regularization on layers with the most significant activation derivatives.