H-Res Adapts Large Transformers Efficiently, Preserving Associative Memory

Large Transformer models function as dense associative memories, retrieving knowledge through complex high-dimensional dynamics. However, adapting these pre-trained models to new tasks presents a fundamental challenge known as the "Plasticity-Stability" dilemma, where modifying the model for new information can lead to forgetting old knowledge. Existing methods either risk catastrophic interference by directly changing model weights, like LoRA, or degrade the model's associative capacity by adding static prompt tokens, as seen with VPT. This research introduces H-Res (Hierarchical Residual Steering), a novel mechanism designed to overcome these limitations. H-Res modulates the effective energy landscape of the Transformer model without altering its global equilibrium or increasing its sequence length. It frames adaptation as a control problem on the activation manifold, learning a state-dependent vector field that guides token trajectories towards task-specific solutions. The study formally proves that H-Res preserves the attention entropy of the foundation model and facilitates neural collapse, a phenomenon where features for the same class converge. Empirically, H-Res demonstrated a 26% improvement over global weight modification on associative retrieval tasks and eliminated the computational overhead associated with prompt-based methods, proving effective in structured domains.