Semantic ID Hierarchy Improves Off-Policy Evaluation for Recommenders.

Generative recommender systems are increasingly using semantic IDs (SIDs), where each item is represented by a hierarchical sequence of discrete codes. Before deploying new recommender variants, teams often rely on off-policy evaluation (OPE) to assess their potential performance without costly A/B tests. This paper investigates whether the recommender model's own SID tree can be effectively used as an action abstraction for OPE. The findings indicate that direct per-item OPE is often unreliable due to insufficient sample sizes in production logs. However, by marginalizing items into code-prefix clusters, which is facilitated by the SID tree, the estimable support for OPE is restored, leading to reduced error. While the gain primarily comes from this coarsening, the SID tree makes this coarsening practical in generative systems because it provides a direct and efficient way to calculate cluster masses. The resolution depth of the coarsening acts as a crucial control knob, allowing for coarser granularity when data support is scarce. A conditional bias bound connects this coarsening bias to the quantizer's reconstruction residual and the target-logging divergence, providing a theoretical understanding of the trade-offs involved.