SceneBot Enables Humanoids to Track Complex Scene Interactions

Current humanoid reinforcement learning policies are adept at free-space movements but struggle significantly with tasks involving physical contact with objects or uneven terrain. This limitation arises because purely kinematic tracking cannot resolve the physical ambiguities inherent in such interactions. Researchers have introduced SceneBot, a unified motion-tracking framework designed to overcome these challenges. SceneBot trains a single policy by providing it with both reference motions and explicit per-link contact labels, which define the expected interactions with the environment. This allows the humanoid to understand and execute complex behaviors involving contact. To address the scarcity of annotated interaction data, SceneBot employs a hindsight scene reconstruction method. This technique infers scene-interaction graphs directly from retargeted human motion. By training on 7.5 hours of this reconstructed, contact-rich data, SceneBot demonstrates the ability to generalize to new motions and environments, performing tasks like carrying a box upstairs, marking a significant step towards general humanoid control.