T-Step Lookahead Improves Whittle Index for Restless Bandits

Restless multi-armed bandits (RMABs) are a powerful framework for sequential decision-making under uncertainty, but solving them, especially under partial observability, is computationally challenging. Whittle index policies offer a scalable approximation, but determining the exact index often requires solving complex infinite-horizon problems. Previous work has introduced linear approximations that simplify this, but these often rely on a one-step active-passive comparison, neglecting longer-term consequences. This research extends the framework by proposing a "t-step lookahead threshold policy." This policy defines the indifference subsidy based on a t-step finite-horizon value iteration, making the threshold subsidy-dependent and more closely tracking the exact decision boundary. The authors prove that this t-step approximate Whittle index converges geometrically to the exact Whittle index. Numerical experiments confirm the method's effectiveness, showing a significant reduction in index error and improved performance over one-step baselines, while maintaining reasonable computational costs.