Training Safe AI Agents Using Human Preferences and Justifications

Training AI agents for safety-critical environments is challenging, especially when environment dynamics are unknown and explicit reward functions are unavailable. Traditional reinforcement learning often falls short in such scenarios. A new human-centered method, DROPJ, proposes a solution by combining learned world models with human feedback. The process begins by training a world model from real-world trajectories, creating a simulated environment. Humans then interact with this simulator, generating informative trajectories. From these, pairs of trajectory segments are presented to humans, who provide preferences and explicit justifications for their choices. This data trains a reward model, which, alongside the world model, enables direct agent deployment via model predictive control. Experiments show that generating informative simulated trajectories significantly reduces training computational costs and improves deployment performance. Furthermore, using preferences over other feedback types, and especially incorporating safety justifications, substantially enhances safety and prioritizes user-defined safety aspects during deployment.