Benchmarking Reveals Pitfalls in Offline Top-k Policy Evaluation

A new study offers a comprehensive benchmark and practical guide for evaluating deterministic top-k allocation policies using offline data, a common challenge for organizations aiming to predict policy performance before deployment. The research systematically investigates the reliability of off-policy evaluation (OPE) methods, uncovering several critical failure modes. The findings emphasize that the trustworthiness of OPE is heavily influenced by the alignment between the logging policy's actions and the target policy's actions, rather than just the logging sharpness. Weak overlap, where the logger rarely takes the actions the target policy would, severely degrades estimates. The study also demonstrates that the "optimizer's curse" persists even with cross-fitting of outcome nuisances, requiring honest policy-level splitting to avoid reuse bias. Furthermore, propensity-estimation error is identified as the most significant source of degradation, impacting Inverse Propensity Score (IPS) estimators profoundly and leaving doubly-robust methods almost unchanged. The research provides a reproducible benchmark and public data, offering practitioners crucial insights into when and how to trust offline evaluations for top-k allocation.