ZeroBN Optimizes DNNs for Edge Latency Constraints

Deploying deep learning applications on edge devices is crucial for addressing privacy and latency concerns associated with cloud servers. However, designing deep neural networks to maximize performance while meeting strict real-time latency requirements on edge systems remains a challenge, as conventional optimization methods don't directly control inference time. This research proposes a novel latency-oriented neural network learning method that utilizes "Zerorized Batch Normalization" (ZeroBN). The goal is to optimize models for high accuracy while strictly adhering to predefined latency constraints. A key innovation is a universal, hardware-customized latency predictor that enables this optimization through a single training process. Experiments show that ZeroBN effectively meets hard latency constraints and maintains high accuracy. For example, it reduced GoogLeNet's latency on an NVIDIA Jetson Nano from 40.32 ms to 34 ms with only a minor accuracy drop, and even improved VGG-19's accuracy on a Jetson TX2 while significantly reducing its latency.