TaskSense Enhances World Models by Focusing on Relevant Visuals.

Traditional world models for visual control often learn latent states by reconstructing entire observations, which can lead to representations being diluted by task-irrelevant background clutter. This inefficiency severely degrades performance, especially in environments with visual distractions. TaskSense addresses this by introducing a task-centric world modeling framework that prioritizes relevant visual information. TaskSense employs a differentiable stochastic spatial attention mechanism, conditioned on the previous latent state, to enforce task relevance *before* latent encoding. Instead of reconstructing the full visual input, the world model only reconstructs the attended regions. An auxiliary inverse-dynamics objective further guides this attention towards control-relevant areas. This approach ensures that latent representations primarily capture essential features, making the model significantly more robust to visual distractions. TaskSense maintains competitive performance on standard control suites and substantially outperforms baselines on distracting environments.