EnerInfer Optimizes On-Device LLM Inference Energy

Deploying Large Language Models (LLMs) directly on devices offers benefits like privacy and reduced costs, but it faces significant challenges related to energy consumption and thermal management. Current systems primarily focus on maximizing decoding speed, often overlooking the potential for energy savings. This research highlights that by slightly reducing NPU and memory frequencies, substantial energy efficiency gains and heat reduction can be achieved without compromising the user's quality of experience (QoE). However, identifying the most energy-efficient configuration is complex, as it varies across models, inference engines, platforms, and runtime conditions. Commercial devices also lack detailed component-level power sensing, and shell temperature is influenced by various dynamic factors. To address these issues, EnerInfer proposes a new framework that moves beyond per-model profiling and heavy sensor reliance. EnerInfer employs disaggregated, model-structure-aware prediction and ranking-driven online feedback. It can predict throughput and power for new LLMs across different NPU/DDR frequency settings, select efficient configurations that meet QoE requirements even under runtime interference, and use lightweight thermal prediction to switch between energy-optimized and thermally constrained modes. Evaluations on real-world LLMs demonstrate significant energy efficiency improvements—up to 65% on phones, 12% on laptops, and 24% on development boards—all while maintaining QoE.