Hindsight Relabeling Boosts Reinforcement Learning Sample Efficiency Fivefold

Reinforcement learning (RL) for vision-language-action (VLA) models often struggles with sample efficiency, as collecting robot rollouts is slow and expensive. When policies are weak, they frequently fail, providing little useful data for learning, even if the failures involve coherent actions. Researchers have introduced Learning from Hindsight (LfH), a technique that applies hindsight relabeling to RL post-training for VLAs. This method reinterprets failed rollouts as successes for different, actually achieved tasks. A single vision-language model is used to relabel both the instruction and the reward, proposing a hindsight instruction for groups of failed rollouts and assessing how well each rollout satisfies it. The policy then trains on both the relabeled and original rollouts. Because VLAs can generalize across language, this linguistic relabeling allows the policy to extract more learning from the same set of trajectories. LfH demonstrated a fivefold improvement in sample efficiency on out-of-distribution LIBERO-PRO tasks, outperforming dense progress-reward baselines and showing consistent gains across different VLA backbones and on physical robots.