Learnijoy
ResearchAI ResearchAI Engineering & DevTools

Weight Direction, Not Magnitude, Drives Grokking in Neural Networks.

Truong Xuan Khanh· July 9, 2026 View original

Summary

This research introduces "cross-trajectory chimera interventions" to study how different components of a neural network's weights contribute to the "grokking" phenomenon. It finds that the direction of weight vectors determines the specific solution a network converges to, while weight magnitude primarily influences the speed of learning.

Researchers investigated the "grokking" phenomenon in neural networks, where models generalize long after overfitting training data. They developed a novel technique called "cross-trajectory chimera interventions," which involves combining the weight norms from one independently trained network with the weight directions from another. This allowed them to isolate the causal roles of these two weight properties. The study revealed that the direction of a network's weight vectors is crucial for determining which specific solution or "circuit identity" the network adopts. When a donor network's weight directions were implanted into a recipient network, the recipient consistently converged to the donor's solution. In contrast, the magnitude of the weights primarily affected how quickly a network learned and how susceptible its learned identity was to being overwritten, rather than dictating the identity itself.

Why it matters

Understanding the distinct roles of weight magnitude and direction provides deeper insights into how neural networks learn and generalize, which could inform the design of more robust and efficient AI models.

How to implement this in your domain

  1. 1Explore architectural designs that explicitly leverage or control weight direction for specific learning objectives.
  2. 2Develop training regularization techniques that prioritize or stabilize weight directions to improve generalization.
  3. 3Investigate transfer learning strategies that selectively transfer weight directions rather than full weights for faster adaptation.
  4. 4Design diagnostic tools to visualize and analyze the evolution of weight directions during training to identify grokking precursors.

Who benefits

AI DevelopmentMachine Learning ResearchSoftware EngineeringAutonomous Systems

Key takeaways

  • Weight direction is a primary determinant of the specific solution a neural network learns during grokking.
  • Weight magnitude mainly influences the learning speed and susceptibility to change, not the solution identity.
  • Cross-trajectory chimera interventions offer a new method for causally dissecting network properties.
  • These findings could lead to more targeted and efficient training and transfer learning strategies.

Original post by Truong Xuan Khanh

"arXiv:2607.06628v1 Announce Type: new Abstract: Which properties of a partially trained network are causally portable to a different, independently trained network? Single-trajectory interventions show necessity within one run, not portability across runs. We introduce cross-traj…"

View on X

Originally posted by Truong Xuan Khanh on X · view source

Want to go deeper?

Turn these trends into skills with Learnijoy's hands-on AI & tech courses.

Explore courses

More in AI Research

AI ResearchAI Engineering & DevTools

Transformers Learn Non-Invertible Modular Multiplication via Stratified Fourier Mechanisms.

This research investigates how small transformers learn modular integer multiplication over composite moduli, a non-invertible operation. It proposes the "monoid extension" theory, suggesting models partition input space into hierarchical algebraic regions where Fourier mechanisms apply, explaining how embeddings, attention, and local features contribute to the computation.

Zitong Andrew Chen, Junaid Hasan, Akhil Srinivasan, Hemkesh Bandi, Jarod AlperJul 9, 2026
AI Engineering & DevToolsAI Research

New Interpretable Model Handles Feature Interactions in Tabular Data.

This paper introduces Interaction Aware Interpretable Machine Learning (IAIML), a framework for tabular data that addresses the limitation of traditional interpretable models in capturing feature interactions. IAIML uses adaptive discretization, pairwise interaction scoring, and a partitioned explanation budget to achieve high accuracy while maintaining interpretability.

Srikumar KrishnamoorthyJul 9, 2026
AI ResearchAI Engineering & DevTools

Principles of Deep Feedforward ReLU Networks Unveiled.

This paper systematically studies the mechanisms of deep feedforward ReLU networks, generalizing principles from two-layer networks to deeper architectures. It explains how hidden-layer units form piecewise linear manifolds to divide input space and how paths and their relationships are central to understanding the back-propagation training solution.

Changcun HuangJul 9, 2026