New Research Shows Shallower ReLU Networks for Max Function

The complexity of representing certain mathematical functions using Rectified Linear Unit (ReLU) neural networks is a fundamental question in deep learning theory. This research focuses on the maximum function, which is a building block for many complex operations. The authors demonstrate a significant theoretical breakthrough by proving that the maximum of up to ten real numbers can be precisely represented by a ReLU network using only two hidden layers. This is achieved through a novel approach that reduces the problem to exact rational linear algebra, leveraging computational verification. A key insight from their construction for the maximum of ten numbers is a structured first hidden layer composed solely of pairwise maximum operations. This structural property allows for recursive application, leading to improved depth bounds for representing the maximum of an arbitrary number of inputs. Specifically, for any number of inputs greater than ten, the maximum function can be represented with fewer hidden layers than prior state-of-the-art methods. These findings extend beyond the maximum function itself. Through the generalized hinging-hyperplane representation, the same improved depth bounds apply to all continuous piecewise-linear functions. This implies that for continuous piecewise-linear functions on up to nine dimensions, a two-hidden-layer ReLU representation is achievable, significantly improving upon previous theoretical limits.