New Distillation Method Trains Compact Instruction-Following Rerankers.

Deploying compact instruction-following rerankers is highly desirable for efficiency, but traditional distillation methods often limit student learning to the teacher's observed ranking space, hindering performance. This research re-examines reranker distillation through a reinforcement learning lens, proposing a more effective two-stage framework. The first stage involves strengthening a larger 4B teacher reranker using off-policy GRPO (Generalized Policy Optimization) with feedback from an LLM-judge on a vast set of instruction-following examples. This enhances the teacher's capabilities beyond its initial training. In the second stage, a compact 1B student model samples rankings from its own policy and receives soft, teacher-derived rewards based on these self-sampled rankings. This innovative coupling of student exploration with knowledge transfer leads to substantial performance improvements, particularly when facing distribution shifts. The resulting 1B reranker achieves competitive performance against much larger models, offering a favorable quality-efficiency trade-off.