Optimal LLM Tokenizer Vocabulary Varies by Deployment Regime

The choice of tokenizer vocabulary size is a fundamental decision in Large Language Model (LLM) infrastructure, typically set based on convention during training. However, new research reveals that the optimal vocabulary size is not constant but rather a dynamic parameter influenced by the LLM's deployment regime, particularly inference volume and serving batch size. The study formalized total deployment cost as a function of training cost, inference cost, inference volume, and batch size. Through controlled experiments on different GPU families (A10G and A100), the researchers observed significant shifts in the inference-optimal vocabulary. For instance, the optimal size varied 16-fold, from 32,000 at a batch size of 1 to 524,000 at batch sizes of 64 or more. This variation is primarily driven by the amortization of the unembedding matrix read, which is a V x d matrix. Furthermore, at model scales of 1.3-2.3 billion parameters, quality (measured in bits per byte) was optimized around a vocabulary size of 65,000, confirming that optimal vocabulary is also scale-dependent. Crucially, the lifecycle-optimal vocabulary can diverge from the training-optimal by up to 16 times for production deployments. The research found that quality remained approximately invariant across the optimal range (less than 2% BPB spread), indicating that vocabulary size can be treated as a pure systems optimization without a quality penalty within this range. These findings offer practical guidance for capacity planning: on-device deployments should target smaller vocabularies (around 32k), while datacenter serving with high inference volume and large batch sizes should opt for larger vocabularies (131k-262k).