New Experimental Design Optimizes A/B Testing for Population Shifts.

When conducting randomized experiments, such as A/B tests, it's common for the experimental population to differ from the target population where the results will be applied. Traditional sample allocation methods can be inefficient in these scenarios, either wasting resources on groups less relevant to the deployment population or under-sampling groups that are statistically challenging to measure accurately. Researchers have developed Target-Weighted Neyman Allocation (TWNA) to address this challenge. This two-stage stratified design uses initial pilot estimates of outcome variances within different groups to intelligently allocate sample sizes and treatment probabilities for the main experiment. The core idea is to balance the importance of a group in the target deployment population with the statistical difficulty of precisely measuring its treatment effect. TWNA offers a robust solution, even when the exact composition of the target population is uncertain. Simulations and benchmarks demonstrate that this method yields significant gains, particularly when certain groups are both highly important for deployment and inherently difficult to measure with precision. This approach ensures that experimental budgets are used more effectively to derive actionable insights for real-world application.