Scaling Laws for Classical ML on Tabular Data.

While scaling laws are well-studied in deep learning, their application to classical machine learning on tabular data has been less explored, often limited to small-scale studies. This research presents a comprehensive, distributed benchmark study involving 127 students, each running a standardized protocol on multiple datasets. The study generated over 11,000 training runs across 18 tabular classification and regression datasets and 6 model families (including Boosting, Random Forest, SVM, and linear models). Three key findings emerged. First, power laws of the form error(N) = a N^(-b) + c consistently fit the learning curves, with tree ensembles generally outperforming linear models at full data. Second, within most model families, approximate shared exponents were observed across different datasets, suggesting a degree of predictive compressibility, though not full universality. Finally, the study highlighted significant replicator-implementation variance, where independent re-implementations of the same protocol, even with fixed random seeds, showed measurable differences in fitted exponents, indicating the impact of unconstrained protocol elements like preprocessing.