Reward Alone May Not Teach Latent State in RL Agents

A fundamental question in reinforcement learning (RL) is whether agents achieving high rewards truly understand the underlying latent state of their task or merely exploit reward-correlated shortcuts. Researchers addressed this by using hidden deterministic finite automata (DFAs) as a transparent testing ground, allowing them to precisely define and observe the "true state." The study found that high reward does not automatically imply latent state learning. Even with strong optimizers like PPO+GAE, state recovery was often partial and highly variable. Task structure, particularly permutation (group-language) properties, was identified as a strong predictor of "perception gaps," where the latent state is representable but not linearly recoverable by the agent. Furthermore, the informativeness of observations played a role in state recovery. This research introduces a crucial distinction: a "perception gap" (latent state not recoverable) versus a "planning gap" (state recoverable but unused for optimal action). This highlights that reward-only evaluation is insufficient for assessing an agent's true task understanding, and that state recovery is predictable in advance.