Counterfactual Estimation Accelerates A/B Tests by Reducing Variance

This research addresses a key inefficiency in standard A/B testing: when control and treatment policies suggest the same action, the resulting outcome adds noise but no useful signal for assessing the treatment effect, thereby inflating confidence intervals. The authors propose a new experimental protocol that exploits this "policy overlap" to accelerate A/B tests. The core innovation involves reframing the randomized treatment assignment as a meta-policy and applying Delta-Off-Policy Estimation methods. This approach yields unbiased estimates for average treatment effects, with the crucial advantage that its variance scales with the divergence between policies, rather than the raw outcome variance. Theoretically, this method recovers standard A/B testing in general cases but strictly dominates the traditional Difference-in-Means estimator whenever policies share common support and the overlap region contributes residual variance. Empirical results confirm these theoretical gains, promising significant improvements for evaluating systems like recommender systems, information retrieval pipelines, and large language model interfaces.