Looped Language Models Face Readout Blind Spot in Dense Supervision.

New research highlights a fundamental challenge in training "looped" language models, which feed hidden states back into future computations. The study reveals that standard dense per-loop cross-entropy supervision often fails to adequately control all internal state variables, specifically the radial scale of hidden states. This occurs because common scale-invariant readouts, like RMSNorm or LayerNorm, effectively hide this scale from the immediate loss function, even as the recurrent network continues to update it. The paper demonstrates that this "readout blind spot" can lead to hidden-state norms growing excessively large in models without inter-loop normalization. To counteract this, the researchers propose two main solutions: either making the scale visible to the loss function through scale-visible readouts or explicit norm penalties, or implementing architectural changes that inherently remove scale from the recurrent loop. These scale-controlled variants show improved perplexity at comparable inference depths.