Spec-AUF Improves Speculative Decoding for LLMs

Speculative decoding is a technique used to accelerate autoregressive language model generation by having a smaller, faster "drafter" model propose a block of tokens, which a larger "target" model then verifies. Only the longest accepted prefix of these drafted tokens is committed. A challenge arises because block drafters are often trained with a full-block cross-entropy loss, supervising every position, even though inference discards tokens after the first rejection. This train-inference misalignment leads to suboptimal drafter performance. Researchers introduce Spec-AUF (Accept-Until-Fail), a novel training objective that tackles this by concentrating supervision solely on the accepted prefix. Unlike other acceptance-aware objectives, AUF is a single, detached modification to the cross-entropy support, requiring no auxiliary objectives, verifier rollouts, or changes to the inference pipeline. Experiments with Qwen3-8B show that AUF significantly increases the average emitted length (τ) of the DFlash drafter across six benchmarks, from 2.40 to 2.61, and similarly improves Domino's two-branch head. This indicates that focusing supervision on the practically relevant part of the block during training leads to more effective drafters for speculative decoding.