New Optimizer RODE Decouples Radial and Directional Neural Network Updates

A novel optimization engine named RODE (Radial-Orthogonal Decoupled Engine) has been introduced, addressing a limitation in current matrix-aware optimizers used for neural network training. Traditional methods often combine changes to a weight's norm and direction in a single step, leading to complex interactions where one can inadvertently influence the other. RODE tackles this by providing distinct update rules and step sizes for the radial (norm) and directional components of the weight matrix. Specifically, RODE updates the matrix Frobenius norm using a scalar radial rule, while its directional channel performs Newton-Schulz-conditioned updates within the tangent space. This decoupling allows for more precise control over the optimization process. Experiments on GPT-2 interventions demonstrated benefits from both direct norm control and RODE's directional updates. Across language modeling and image classification tasks, RODE consistently outperformed existing Muon variants, achieving lower loss and significantly reduced final global model norms. For instance, on a 1.5B scale language model, RODE lowered loss from 4.145 to 3.346 and reduced the final global norm from 11964 to 2183 compared to Muon RMS. It also showed superior performance in fine-tuning Qwen3.5-9B, suggesting that separating radial and directional dynamics offers a more effective and controllable approach to matrix optimization.