Hybrid Decoding Strategy Reveals LLM Evaluation Challenges.

This paper introduces Speculative Refinement (SpecRef), a novel, training-free hybrid decoding strategy that merges autoregressive (AR) and diffusion language models. SpecRef works by using an AR draft to warm-start a masked diffusion model, employing entropy-guided selective masking for refinement. The researchers applied SpecRef across various benchmarks, including code generation and general reasoning tasks, to understand how combined generation systems should be evaluated. Their analysis uncovered several critical insights into current evaluation practices. They found that code benchmarks often confuse a model's ability to generate correct syntax with its logical correctness, as providing a syntactic scaffold dramatically improved accuracy without changing the underlying model. Furthermore, a "refinement tension" was observed where multi-stage correction processes could inadvertently degrade already correct tokens, exposing limitations in single-model evaluations. The study also noted discrepancies between log-likelihood and generative evaluations, suggesting they measure different aspects of model capability, and identified issues with standard Python post-processing for non-AR generators. These observations are crucial for anyone developing or evaluating multi-stage or non-autoregressive generation pipelines.