Layerwise Progressive Freezing Improves Deep Binary Neural Network Training.

Training binary neural networks (BNNs) from scratch often leads to significant accuracy loss as the networks become deeper. This issue is largely attributed to the straight-through estimator (STE), a common technique used in BNN training, which struggles with the mismatch between its forward and backward passes. This research explores a different angle: the timing and location of binarization during the training process. The paper introduces StoMPP (Stochastic Masked Partial Progressive Binarization), a novel training scaffold. StoMPP gradually replaces continuous weights and activations with their hard binary counterparts, layer by layer, moving from the input to the output. It uses stochastic partial masks and soft refreshing to manage this progression. This method offers two key advantages: as a standalone, STE-free training procedure, it substantially improves accuracy over vanilla STE, with benefits increasing with network depth. For instance, it shows significant gains on ResNet-50 BNNs across various datasets. When StoMPP is combined with surrogate gradients, applying STE only to the frozen binary entries, the performance gains are even more pronounced. A core finding is that the order of progression is critical: forward layerwise progression prevents depth collapse, while reverse progression leads to near-chance performance. This asymmetry is traced to activation-induced gradient blockades, where committed binary activations sever upstream gradient flow, and the progression order controls when these blockades form.