Neural Network Nonlinearity Reimagined as Input-Conditioned Threshold Gating

Traditional neural networks rely on activation functions to introduce nonlinearity, enabling them to approximate complex functions. This research introduces a new concept called Threshold Gating (TG), which posits that this essential nonlinearity can be achieved through input-conditioned thresholding across different branches. The study demonstrates that common activation functions, both piecewise-linear like ReLU and smooth ones like SiLU, can be precisely represented as instances of this TG primitive. The researchers validated their findings by converting various pretrained models, including CNNs, transformers, and recurrent networks, to use the TG primitive without any loss in performance or the need for retraining. Beyond mere conversion, training models from scratch with TG can lead to advantages such as improved model compression, enhanced performance, and reduced training times. The paper also discusses the hardware implications, noting that TG offers a unified implementation for analog in-memory systems, potentially alleviating bottlenecks related to analog-to-digital converters.