Two-Hidden-Layer ReLU Networks Can Represent MAX Functions.

This research investigates the ability of two-hidden-layer ReLU neural networks to precisely represent the MAX function, which calculates the maximum value among N inputs. This problem is fundamental to understanding the architectural complexity required for neural networks to model continuous piecewise linear functions. Previously, the exact representation for MAX_N was an open question for N greater than 5. Through a meticulous computer-assisted search, the authors successfully derived two-hidden-layer ReLU representations for MAX functions with N values up to 8. Their method involves constructing rational linear combinations of terms, where inner maximum operations are handled by the first hidden layer and outer maximums of sums are handled by the second. This finding provides concrete examples that push the boundaries of what is known about network depth requirements. While similar independent work has recently shown representations for N up to 10, this paper offers alternative constructions and techniques that may be valuable for future theoretical advancements in neural network architecture.