Mirror Learning: Acquiring Policies from Third-Person Observation

This paper introduces "mirror learning," a novel framework for imitation learning that allows AI agents to acquire actionable policies by passively observing demonstrations from a third-person perspective. Traditional behavior cloning (BC) typically requires dense, first-person data, which is often difficult and costly to collect. Mirror learning addresses this by leveraging the rich observational signals available in third-person views, much like humans and animals learn. The method involves two key components: first, a learned perspective transformation, achieved using a fine-tuned video diffusion model, which effectively places the learner "in the demonstrator's shoes." Second, an inverse dynamics model inferring the action trajectories corresponding to the observed behavior within the learner's own control space. This process synthesizes "mirror data," which is essentially pseudo first-person expert data generated from third-person observations. Empirical results show that policies trained solely on mirror data can be effective, and augmenting first-person BC training with mirror data further enhances performance. This suggests that modern generative world models implicitly contain enough structure to enable a scalable and safer alternative to data collection methods reliant on extensive teleoperation.