Optimizing Comparison Pair Selection for LLM Post-Training.

The field of large language model (LLM) alignment heavily relies on preference-based post-training, which typically involves generating multiple completions for a prompt and then labeling comparison pairs based on human preferences. Given that human labeling is often a costly process, researchers are exploring ways to optimize the use of labeling budgets. Instead of labeling all possible pairs from a small set of completions, a more efficient approach might be to generate a larger pool of completions but selectively label only the most informative comparison pairs. This study frames the problem of comparison curation as a sampling-design challenge, aiming to identify which pairs, when labeled, lead to the highest quality final policy after post-training. Focusing on Direct Preference Optimization (DPO), the research provides theoretical bounds on the optimality gap, showing that comparison selection impacts performance through a specific information matrix. This insight leads to an explicit optimization criterion for budgeted comparison curation and suggests practical sampling designs that consistently improve sample efficiency compared to common heuristics in both synthetic and real-world LLM post-training benchmarks.