New Method Detects and Adjusts for Temporal Leakage in LLM Backtests

Evaluating large language models (LLMs) often involves backtesting, where performance is assessed on historical data. A common practice to check for data contamination, or "temporal leakage," is to compare scores before and after a training cutoff date. However, this research reveals that this standard check is ineffective. The study shows that LLMs legitimately perform better on information closer to their training cutoff, making recency appear like leakage, and it's impossible to distinguish the two passively. To address this, the paper introduces two novel methods for measuring and adjusting for temporal leakage. The first uses a known cutoff to pinpoint leakage at the boundary of the training data. The second employs a matched clean control to identify global leakage, providing a leakage-adjusted score. These estimators were validated by intentionally injecting leakage into twin models, successfully recovering the injected dose and confirming null results on clean questions. Applied to frontier models, these methods detected specific cutoff-localized leakage and cleared several models whose apparent advantages were solely due to recency, emphasizing the need for a more robust backtesting approach.