TUP Distillation Improves LLM Generation by Truncating Low-Ranked Candidates

The research proposes TUP, a novel distillation policy designed to improve the quality of generated content from large language models. Current methods like Best-of-N sample multiple candidates and use a reward model to select the best, with distillation aiming to embed this selection process into a single policy. Existing rank-based distillation often uses smooth reweighting, which reduces the mass of low-ranked completions but keeps them in the target support. TUP distinguishes itself by actively removing low-ranked completions from the support entirely, focusing only on reweighting the retained upper tail with adjustable sharpness. This approach is theoretically supported, showing that under certain conditions, the best monotone rank-reweighting can be achieved by a lower-tail truncation rule. The policy can be trained entirely offline using binary cross-entropy, leveraging shifted-truncated win-rates as soft labels. Empirical results indicate that TUP is competitive with strong offline alignment baselines, offering a more decisive way to refine generation policies.