Diffusion Language Models Encode Latent Denoising Progress

Diffusion Language Models (DLMs) are emerging as a promising alternative to traditional autoregressive models. Unlike standard diffusion models that are explicitly conditioned on a timestep to guide their denoising process, DLMs do not have this explicit conditioning. This raises a fundamental question about whether these models implicitly represent their denoising progress and how such information might be utilized internally. This research demonstrates that DLMs do indeed encode a latent representation of the diffusion timestep within their internal residual streams. This "subliminal clock" signal can be reliably extracted using probing techniques across different layers of the model, indicating that the denoising progress is decodable from the model's internal activations. Furthermore, by steering the model along a specific low-dimensional subspace associated with this inferred timestep, researchers found they could systematically modulate the model's confidence and entropy in its outputs. The study also analyzed the geometric properties of this identified representation, revealing that it exhibits structured and interpretable characteristics within the activation space. These findings provide crucial insights into the internal workings of DLMs, explaining how they manage and process information related to their generative process, even without explicit external guidance. Understanding this latent time modeling can inform future advancements in DLM design and control.