New Method Improves Cross-Tokenizer Distillation for LLMs

Open-weight language models from different families often possess complementary strengths, making their consolidation into a smaller, more efficient student model through on-policy distillation (OPD) highly desirable. A significant challenge in this process arises when teacher and student models use different tokenizers, as existing cross-tokenizer methods can lead to information loss or misattribution of probabilities. To address this, a new technique called Byte-Prefix Marginalization (BPM) has been developed. BPM re-expresses the teacher's next-token distribution over the student's vocabulary by operating in a shared byte space. It assigns each teacher token's probability to the longest student token whose byte representation is a prefix of the teacher token's bytes, aggregating probabilities and handling unmatched mass explicitly. This approach creates a vocabulary-complete, byte-aligned, and mass-preserving target distribution for dense OPD. BPM has been shown to consistently outperform current cross-tokenizer methods across various mathematics and programming benchmarks, achieving significant improvements in average performance. This innovation allows for more effective knowledge transfer from diverse large language models to smaller, specialized student models.