Second-Order Theory-of-Mind Improves Human-AI Team Alignment

This research re-conceptualizes preference-based reward learning, typically used to align robot and agent behavior with human intent, as a collaborative human-autonomy team problem. Traditionally, the human teacher is treated as a passive oracle. However, this work argues that an informed teacher, possessing knowledge of the objective, can construct training examples far more efficiently than any learner-driven strategy, especially as the complexity of the reward function increases. To exploit this advantage, the teacher needs an accurate model of what the learner currently knows. The proposed solution involves a dual-model approach: the human teacher maintains a model of the learner to design an effective curriculum, while the AI learner maintains a "second-order model" of the teacher's model. This second-order model allows the learner to generate "understanding statements" – structured preference constraints – which help keep the teacher's model of the learner synchronized and accurate. Simulations demonstrated that an informed teacher significantly outperforms learner-led selection. Crucially, while teacher-model drift under alternating teachers can erode this advantage, the use of understanding statements, particularly second-order (ToM-2) statements, effectively repairs this drift. ToM-2 statements proved superior when the teacher's error about the learner was concentrated rather than evenly spread, leading to better belief synchronization and overall team performance.