Training Under Challenge: Certifying Neural Network Optimality

When training neural networks, a flat training curve doesn't definitively indicate whether a global optimum has been reached, or if the model is merely stuck in a local minimum, limited by its representation, or mismatched to its trainer. To address this ambiguity, researchers propose "Training Under Challenge," an executable-certificate framework. This framework involves predeclared, architecture-valid procedures that construct complete alternative solutions within the same certified class and reevaluate the same objective function. Any candidate that yields a lower objective value serves as a replayable witness, providing a lower bound for the checkpoint's empirical global-optimality gap. While passing a finite suite of challenges offers suite-relative conclusions, global-gap conclusions require a separately justified coverage mechanism. The research defines a resource-indexed challenge-power modulus that characterizes the largest gap compatible with passage. For squared loss, current block-decrease operators enable coverage checking and yield uniform and realized-residual bounds. A key theoretical finding is the "converse frontier," proving that without coverage, a first-order ReLU trainer can converge to a non-global point while achieving infinitely many exact conditional head optima. Experiments on a channel-gated ResNet-18 distillation problem showed that eight internal challenges covered all 240 audited output directions, with realized-residual bounds within factors of 1.74–3.02 of the true gap. Paired predictive certificates also help diagnose decoder under-use versus representation insufficiency, and quantized-denoising studies demonstrate diagnosis, repair, and recertification capabilities.