Fractional Optimizers and Fractal Activations Improve Neural Networks

This research explores the synergistic potential of two distinct neural network advancements: fractional optimizers and fractal activation functions. Fractional optimizers enhance traditional first-order methods by incorporating fractional derivatives and memory effects, while fractal activations introduce multi-scale, non-linear representations inspired by self-similar functions. The study systematically evaluates their interaction across various scenarios, including benchmark optimization surfaces (some perturbed with fractal noise) and feed-forward neural networks applied to ten classification datasets. The comparison includes a range of optimization techniques, from standard methods to regularization-style and adaptive memory-based fractional optimizers. Key findings indicate that while both approaches offer benefits, their optimal pairings are selective. Regularization-style fractional scaling shows strong performance with certain fractal activations in network training, whereas Gr"unwald--Letnikov memory proves more effective on perturbed surfaces. Adaptive memory variants generally improve upon plain memory substitution, suggesting controlled fractional memory is a promising, though not universally superior, direction.