New Framework Clarifies World Models for Reinforcement Learning

This research introduces a refined conceptual framework for understanding "world models," a core component in model-based reinforcement learning. Instead of solely focusing on what variables world models predict (e.g., observations, rewards), the paper argues for a prior distinction based on *which channel* they model: the environment, the agent, or the combined agent-environment system. Leveraging computational mechanics, the authors define canonical predictive models for each of these three cases using $\epsilon$-transducers or $\epsilon$-machines. This approach recovers standard predictive state representations for environment models and provides analogous canonical models for the agent and the joint system. A key structural finding is that canonical support-restricted environment states, induced by closed-loop coupling, factor through the canonical joint causal states, with their transition structure derived directly from the joint model. The framework demonstrates its utility with a POMDP/controller example where an unrestricted environment model has infinite states, but the canonical support-restricted model, induced by coupling, is finite. This work clarifies the distinct roles and structures of various world models, offering insights into how coupling and support restriction can fundamentally alter their predictive structure and complexity.