CaRE Protocol Standardizes Evaluation for Masked Diffusion Language Models

A new evaluation framework called CaRE (Compute-aware Remasking Evaluation) has been developed to standardize the assessment of Masked Diffusion Language Models (MDLMs). The rapid advancement of MDLMs has outpaced their evaluation standards, leading to inconsistent and incomparable results across different research papers due to variations in step counts, metrics, and sampling temperatures. CaRE addresses this by standardizing the actual number of function evaluations (NFE), enforcing multi-metric reporting, and explicitly controlling stochasticity. Applying CaRE to seven remasking strategies across various MDLMs revealed critical insights. It demonstrated that temperature significantly influences MAUVE variance and that compute-matched comparisons often reverse previously published strategy rankings. Furthermore, it highlighted a tension between informed remasking and stochastic unmasking, where high-entropy remasking can reduce MAUVE scores. These findings underscore that current MDLM evaluations can mistakenly attribute algorithmic improvements to uncontrolled factors like compute resources and stochasticity, making true progress difficult to ascertain. The release of the CaRE protocol, implementation, and a leaderboard aims to ensure future remasking claims are reproducible and genuinely comparable.