EntropyMoE Boosts Tokenizer-Free LLMs with Sparse Expert Routing

Recent advancements in byte-level large language models (LLMs) have made tokenizer-free modeling increasingly competitive by dynamically grouping bytes into patches. However, existing architectures typically apply the same dense computational resources to every patch, failing to adapt to the diverse semantic and granular variations among patches. This uniform computation limits efficiency and performance. To address this, researchers introduce EntropyMoE, a Mixture-of-Experts (MoE) architecture specifically tailored for dynamic byte patches. EntropyMoE replaces dense feed-forward modules with Top-K expert layers, where each dynamic patch serves as the unit for expert routing. Crucially, the router selects experts based directly on patch entropy, leveraging the same granularity signal used for dynamic patch construction to organize sparse computation. Patch entropy and length together define the feature space for regulating expert specialization. Experiments show EntropyMoE achieves superior bits-per-byte performance while maintaining strong downstream accuracy, establishing patch entropy as an effective routing mechanism for sparse conditional computation in tokenizer-free LLMs.