Exact Network Surgery: Function-Preserving Growth for Neural Networks

This paper formalizes "Exact Network Surgery," a novel technique for dynamically expanding the capacity of neural networks without altering their learned function. Unlike previous methods that might introduce numerical perturbations or require a complete rebuild of the training program, this approach allows for the bit-exact insertion of a residual block into an active computational graph. The method ensures that the network's function remains precisely the same after the insertion, even under specific floating-point assumptions. Crucially, the newly inserted parameters are immediately trainable, and the system is designed to only recompute the necessary downstream parts of the graph, leaving other nodes and optimizer states untouched. The research includes proofs for an identity-morphism theorem for gated residual blocks and a structural-locality theorem, demonstrating the efficiency of the reactive invalidation engine. Empirical validation using NeuroDSL confirms the bit-exactness of grafting, the immediate trainability of new parameters, and the minimal computational overhead, presenting a significant advancement in flexible neural network architecture design.