AI Efficiency: FLOPs Don't Tell the Whole Story

This research undertakes a replication study to re-evaluate the relationship between Floating Point Operations (FLOPs) and actual execution time in AI efficiency assessment. The original study proposed an alpha-FLOPs estimation formula, and this replication aimed to verify its applicability on contemporary, more powerful hardware. During the replication process, significant limitations were encountered in the original study's materials, including a lack of specific dependency details and transparency regarding regression data. Despite these challenges, the replication validates the core thesis: raw FLOPs alone are insufficient for predicting execution time, primarily because different operations exhibit varying degrees of parallelization. However, the study also uncovers new complexities. Fine-grained measurements on newer hardware reveal instabilities and discontinuities in execution time, such as jumps and oscillations, which the original alpha-FLOPs formula generally underestimates. While the empirical findings of the original study regarding FLOPs' inadequacy are confirmed, the alpha-FLOPs estimation itself shows negative results. The paper concludes by emphasizing the crucial need for complete and accurate replication packages in hardware-dependent efficiency research, providing its own comprehensive package.