MoPLEx Learns Plackett-Luce Mixtures for AI Alignment

The problem of learning mixtures of Plackett-Luce models from multi-way ranking data is crucial for AI alignment and preference optimization, especially when annotators exhibit diverse preferences. Previous work on Bradley-Terry models from pairwise comparisons faced identifiability limitations when the number of mixture components exceeded half the ranking length. This research proposes an efficient solution called MoPLEx (Mixtures of Plackett-Luce models with Expectation-maximization). MoPLEx overcomes these limitations by first augmenting rankings to a larger size using a base language model, then employing a gradient-based estimation method to reduce inference costs in the input embedding space. These two steps form the core of its expectation-maximization algorithm. Extensive experiments demonstrate MoPLEx's effectiveness: it estimates true probabilities with high accuracy even for large language models and significantly improves clustering and ranking accuracy (by 43.7% and 15.2% respectively) compared to baselines on preference optimization datasets. This highlights MoPLEx's capability in handling heterogeneous preferences by measuring alignment between gradients.