LLM Scaling Exponents Indicate Unsustainable Energy Use

The current scaling exponents observed in large language model (LLM) applications suggest an unsustainable trajectory concerning energy consumption. This research delves into the reasons behind these small exponents, which imply that the computational resources required to achieve incremental performance gains are growing at an alarming rate. The paper further examines the argument that this "smallness" might be a numerical bias, often attributed to a "pedestal effect" where a non-zero loss function value is neglected in the theoretical limit of infinite data. However, the authors demonstrate that even when this pedestal effect is considered, the fundamental issue of unsustainability in energy resource usage for LLM scaling remains unresolved. Finally, the study draws an analogy with phenomenological models of fluid turbulence to comment on how the smoothness or roughness of the training data can influence these critical scaling exponents. This connection highlights the complex interplay between data characteristics, model architecture, and the efficiency of LLM development, urging a reevaluation of current scaling paradigms.