Frontier AI Forecasting Lacks Robust Measurement, Hindering Accurate Predictions.

Current methods for forecasting the progress of frontier AI systems face significant challenges due to a lack of consistent and comprehensive measurement data. An audit of public records reveals that key metrics like training compute are often missing, especially for closed systems, making it difficult to establish clear connections between resource investment and capability milestones. Furthermore, the evolution of benchmarks creates discontinuities, making direct comparisons and trend analysis problematic. The study emphasizes that a substantial portion of quantitative events originate from a single measurement program, indicating a concentration of provenance. This reliance on limited data sources and the absence of standardized, versioned measurement systems undermine the reliability of current AI progress predictions. The authors suggest that robust forecasting necessitates a more rigorous approach, focusing on explicit measurement systems with clear protocols and source dependencies, rather than merely extrapolating from existing curves.