Internal Pluralism Challenges Pairwise Comparisons in AI Alignment

Standard methods for aligning AI decision rules with human preferences often rely on local pairwise comparisons, assuming these comparisons sufficiently capture an individual's desires and can always be answered decisively. However, new research challenges these assumptions by introducing the concept of "internal pluralism," where individuals evaluate decision rules based on multiple, potentially conflicting, authoritative priorities. The study presents a formal model demonstrating two key failures of forced local pairwise comparisons. Firstly, certain priorities like proportionality or egalitarianism are inherently global, meaning their implications in one scenario depend on broader contexts, which local comparisons fail to capture. Secondly, strong internal conflicts between priorities can force individuals into distorted choices when indecision is not allowed. The findings suggest that allowing people to report indecision can significantly improve the accuracy of preference learning and points towards new methods that directly elicit these underlying priorities for more faithful and interpretable AI alignment.