LoRA Monitors for Diffusion LMs: Top-1 Fails, Max Gradient Succeeds.

A new study explores the effectiveness of diagnostic monitors for fine-tuning discrete diffusion language models (DLMs) using Low-Rank Adaptation (LoRA). The research specifically tested whether top-1 argmax concentration, a common metric, could reliably signal training collapse. Across hundreds of LoRA/PEFT configurations, this monitor consistently fired warnings, yet no actual collapses occurred, indicating zero precision. This failure is attributed to pre-equilibrium saturation, where top-1 concentration becomes high and insensitive early in training. The researchers then evaluated the maximum LoRA gradient norm as an alternative, parameter-side signal. This metric, which samples gradient routing, proved significantly more effective. On a held-out dataset, a train-optimized threshold for max gradient norm achieved a precision of 0.68 and an F1 score of 0.79 in identifying top-decile final-loss configurations. The findings suggest that top-1 concentration should be discarded as a PEFT alarm for DLMs. Instead, logging the maximum gradient early in training and calibrating thresholds per DLM family is recommended for more accurate inspection of training stability.