Rethinking World Model Evaluation for Decision-Making.

The concept of "world models" in AI has expanded significantly, now encompassing various types of models from action-conditioned environment models to synthetic-data engines. This broadening definition has led to a diverse range of evaluation metrics, often resulting in a disconnect where papers make strong claims about a model's utility that aren't fully supported by the evaluation methods used. This paper surveys the current literature and posits that the most critical aspect of evaluating a world model, especially for embodied decision-making, is its ability to support reliable counterfactual reasoning, policy evaluation, planning, and optimization. This includes assessing its performance under interventions, policy-induced distribution shifts, and long-horizon rollouts, rather than solely focusing on visually compelling video generation. To address this, the authors propose an L0-L7 evaluation ladder. Lower levels (L0-L3) focus on diagnostics of generated artifacts like visual plausibility, while higher levels (L4-L7) emphasize genuinely interventional tests and direct evidence of decision-making usefulness, such as policy optimization utility. This framework aims to foreground metrics like counterfactual action fidelity, closed-loop rollout validity, reward/value prediction, and model exploitability, providing a more robust and relevant benchmark protocol for world models.