Neural Networks Achieve Optimal Tradeoff in Single-Index Models

A fundamental question in machine learning is whether neural networks, when trained with gradient-based methods, can achieve the best possible computational-statistical tradeoff for learning Gaussian single-index models. Previous work established a lower bound for polynomial-time algorithms under the statistical query (SQ) framework, but it was unclear if neural networks could meet this sample complexity. This research proposes a unified gradient-based algorithm for training a two-layer neural network. This method is versatile, supporting various loss and activation functions, and is shown to learn a feature representation that strongly aligns with the unknown signal. The algorithm achieves a sample complexity that matches the SQ lower bound for all generative exponents, up to a polylogarithmic factor. Furthermore, the approach is extended to handle sparse underlying signals using a novel weight perturbation technique, also matching its corresponding SQ lower bound. This framework suggests new gradient-based solutions for other challenging problems.