New Framework Accelerates On-Device Diffusion LLM Inference on Mobile NPUs.

Diffusion Large Language Models (dLLMs) are promising for mobile inference due to their parallel token denoising capabilities, which can reduce latency. However, deploying them efficiently on smartphones with Neural Processing Units (NPUs) presents several challenges. These include managing shrinking workloads as tokens are committed, optimizing KV cache reuse, and mitigating costly data transfer and remapping due to limited NPU-visible memory. Researchers have introduced llada.cpp, the first NPU-aware inference framework specifically designed to accelerate dLLMs on smartphones. This framework employs three key techniques: Multi-Block Speculative Decoding to fill NPU workloads, Dual-Path Progressive Revision to handle token stability without stalling NPU execution, and Swap-Optimized Memory Runtime to reduce memory overheads. Evaluations across various hardware and dLLM workloads show that llada.cpp can reduce LLaDA-8B generation latency by 17x-42x compared to CPU baselines, all while maintaining generation quality. This represents a significant leap towards practical, high-performance dLLM inference directly on mobile devices.