LLM Energy Consumption Varies by Attention Architecture

Researchers conducted a detailed empirical study to understand the energy consumption of large language models (LLMs) during the decode phase, focusing on how different attention mechanisms impact energy use. The study evaluated models employing Multi-Head Attention (MHA), Grouped Query Attention (GQA), and Grouped Query Attention with Sliding Window Attention (SWA) under various conditions, including different context lengths, batch sizes, and generation workloads. GPU energy consumption was precisely measured using NVIDIA hardware counters. The findings highlight that the choice of attention mechanism is the most critical factor determining how decode energy scales with context length. MHA models exhibited a significantly steeper increase in energy consumption as context length grew, whereas GQA models showed more moderate growth. Notably, GQA with SWA maintained nearly constant energy consumption regardless of context length. The study also confirmed that overall model size dictates absolute energy consumption, and batching effectively reduces both energy per generated token and request latency by up to 87%. These insights offer practical guidance for selecting energy-efficient LLM architectures and optimizing inference configurations.