Training Logs Can Improve AI Model Comparison Precision.

When comparing different AI models trained stochastically, it's crucial to accurately estimate performance differences and their associated uncertainties, typically requiring multiple runs. This paper investigates whether leveraging data from training logs generated during these runs can enhance the precision of such comparisons. The methodology involves arm-specific covariate adjustment, where each model's performance is adjusted using statistics solely from its own training logs, while the raw mean difference remains the primary reported effect. A study involving three vision architectures and datasets demonstrated that straightforward adjustments based on early training logs frequently led to a reduction in the uncertainty of model comparisons. However, a significant challenge identified was covariate selection; broadly searching through the log data for the most correlated statistics often introduced more noise than it removed, even when potentially useful statistics existed. This suggests that while training logs hold promise for more precise model comparisons, careful and targeted covariate selection is essential to avoid adding noise.