DMAPO Improves LLM Alignment with Less Data

Traditional preference optimization for large language models (LLMs) often focuses on varying training objectives while keeping the dataset fixed. This research, however, explores whether a smaller, highly confident set of on-policy responses can provide a more reliable learning signal. They introduce DMAPO (Data-centric Multi-evaluator Agreement for Preference Optimization), a method that generates candidate responses from a target policy and then rigorously evaluates them. DMAPO employs multiple specialized evaluators to assess helpfulness, factuality, and conciseness, applying a process-critic correction before retaining only high-consensus desirable or undesirable examples. For instance, it accepted only 1,871 out of 54,236 Mistral-7B candidates (3.45%). Training KTO on this refined dataset led to significant performance improvements, including a 7.50 MT-Bench score, a 95.5% length-controlled win rate against a reference model, and 57.3% IFEval prompt accuracy. Independent evaluations by GPT-4o and Claude Opus 4.7 also strongly favored DMAPO over other methods, demonstrating substantial net win rates on both held-out and out-of-distribution prompts. The study also found that while changing the evaluator model or rubric alters the selected examples, it has minimal impact on downstream performance, highlighting the robustness of the consensus filtering approach.