AI Metric Choice Impacts Future Model Capability Distribution.

The ongoing exponential scaling of computational power in AI raises a critical question: will advanced AI capabilities remain exclusive to well-funded entities, or will they become broadly accessible to developers with limited budgets? This research, building on previous work, posits that the answer hinges on how AI capabilities are valued and measured. The paper differentiates between various performance metrics, demonstrating that while some, like validation loss, show a shrinking gap between frontier and smaller models, others indicate that frontier models continuously extend their lead. It provides precise mathematical conditions to determine which functional forms of performance metrics, in relation to training and inference compute, inherently favor "meek" (smaller, less resource-intensive) models. Bounded performance metrics, for instance, consistently favor these more accessible models. However, the interpretation of these metrics is crucial. The research highlights that many common bounded metrics have closely related unbounded counterparts, and vice versa. The choice of an appropriate metric for a given domain is therefore a prerequisite for policy decisions. If a critical capability, such as software engineering or synthetic biology, is measured by an unbounded metric, then frontier-level capabilities are likely to remain concentrated among a few wealthy actors. Conversely, if the relevant metric is bounded, these advanced capabilities could proliferate more widely.