Hidden Boundary Motion Impacts Transformer Optimization

A new study identifies a previously unrecognized phenomenon in Transformer optimization: "hidden boundary motion." This refers to a component within weight updates in affine layers that behaves functionally identically to a bias update, particularly when the input to the layer has a non-zero mean. This hidden contribution, often much larger than explicit bias updates, significantly influences how models learn. Researchers decomposed updates into a sample-varying "shape" component and a shared "boundary" component. They found that under optimizers like AdamW, this bias-like boundary motion is predominantly realized through the weight matrix rather than the explicit bias. A diagnostic optimizer, SBO-AdamW, was developed to orthogonally optimize these components, leading to notable improvements in validation accuracy for a Transformer trained on IMDb. While SBO-AdamW showed promise, it also highlighted challenges like bias-coordinate drift, indicating that a stable centered-affine parameterization is the crucial next step for practical application.