New Protocol Certifies Forecast Models for Deployment Decisions

Forecasting leaderboards typically rank models based on their predictive accuracy, often leading to the assumption that the top-ranked model is automatically the best choice for deployment. However, this research highlights a critical disconnect: a model that performs best on a leaderboard might not be optimal when integrated into a real-world decision interface, such as an alert threshold or a budget allocation policy, especially when factors like switching costs are involved. The study proposes a "fail-closed certification protocol" designed to rigorously evaluate whether a forecast-side winner is truly deployment-actionable for a specified interface and utility. This protocol establishes sufficient evidential conditions to identify situations where a forecast winner could be deployed-suboptimal due to friction or other real-world constraints. Using the Traffic-Hourly dataset as a certified anchor, the research demonstrates that while winners may agree at zero friction, positive switching friction can render the forecast winner suboptimal in deployment. The protocol includes a locked native audit, which successfully blocked 155 apparent forecast/deployment winner inversions across numerous candidates and scenarios, preventing overclaiming. This work emphasizes the need for a conservative approach to translating leaderboard success into deployment decisions.