New Method Stabilizes Deep Equilibrium Model Training

Deep Equilibrium Models (DEQs) are powerful, but their training can be unstable. This instability arises because the implicit differentiation used for training relies on an adjoint system, and if the Jacobian within this system is nearly singular, small input perturbations can lead to greatly amplified, highly sensitive gradients. This makes optimization unreliable. The proposed solution, Response Renormalization, is a backward-pass technique that specifically targets and lifts these problematic near-pole denominators in the Jacobian, without affecting other well-conditioned response channels. The method, particularly its Collective Mode Response Renormalization (CMR) and Phi-adaptive CMR variants, applies this correction in a low-dimensional critical subspace. Extensive testing across 23 multiphysics families, including PDEs and complex geometries, shows that CMR and Phi-CMR achieve test errors within five percent of models trained with exact implicit differentiation in over 98% of static and 95% of transient comparisons. This demonstrates that selective response renormalization effectively controls adjoint amplification, leading to more reliable parameter updates while preserving crucial gradient information for learning.