New Method Speeds Up Visual AI Model Training

Training compact visual autoregressive models often involves a technique called visual on-policy distillation (OPD), where the student model learns from trajectories it generates itself. A major bottleneck in this process is the token-by-token, autoregressive decoding required for these online rollouts, which adds substantial computational cost to each training step. Speculative Jacobi Decoding (SJD) offers an alternative by processing multiple tokens in parallel without needing an additional draft model, but its original design was for single-sequence inference. This research introduces HB-SJD (Batched Speculative Jacobi Decoding), which adapts SJD for batched rollouts in visual OPD. HB-SJD enables each image in a batch to progress independently based on its own decoding status, while still verifying tokens in batched model forwards. It dynamically switches between Full and Compact execution modes as images complete their decoding, further reducing costs in later rollout rounds. By replacing only the student rollout backend and leaving the teacher model and distillation objectives unchanged, HB-SJD has been shown to substantially decrease both rollout and overall training time for models like LlamaGen, without compromising the quality of the distilled student's generation.