Functional Reconstruction Improves MLA Draft Models for Speculative Decoding

Multi-head Latent Attention (MLA) is gaining traction for long-context Large Language Model (LLM) inference due to its efficiency in reducing memory traffic by using compact latent states instead of a growing Key-Value (KV) cache. However, many powerful open-source LLMs utilize Multi-head Attention (MHA) or Grouped-Query Attention (GQA), requiring conversion to MLA for these efficiency benefits. This conversion, particularly when combined with speculative decoding, can lead to a sharp reduction in draft token acceptance, undermining the acceleration benefits. The issue stems from attention-function errors introduced during direct MHA/GQA-to-MLA conversion, which, while potentially tolerable for standalone generation, significantly impact the agreement between draft proposals and target verification in speculative decoding. To counter this, a new method called Functional Reconstruction is introduced. This end-to-end procedure optimizes each converted MLA attention module to precisely reproduce the post-output-projection response of its original MHA/GQA counterpart on calibration hidden states. Functional Reconstruction is converter-agnostic and does not require verifier logits or supervision, preserving the converted cache and inference graph. Extensive evaluations across various model families, converters, backends, and tasks demonstrate that this method materially improves draft token acceptance in a significant majority of cases, making speculative decoding with MLA more effective.