Ternary LLMs Show Stratified Capability Loss, Retain Adaptability

Extreme low-bit inference, particularly ternary quantization (restricting weights to -1, 0, or +1), offers a promising path to create smaller, more deployable language models. A study explored the impact of converting Qwen3.5-0.8B to ternary weights, resulting in a model named Cloe, which underwent 72.4 million tokens of quantization-aware training. The primary question was not just if weights could be quantized, but which capabilities would survive and if the model remained useful for adaptation. The evaluation across 29 benchmarks and representation diagnostics revealed a non-uniform degradation of capabilities. Specialist factual information, for instance, was largely lost, with a linear probe recovering significantly less MMLU answer accuracy from Cloe's representations compared to the full-precision teacher model. However, Cloe retained measurable performance on ten tasks, averaging 77.1% of the teacher's performance. Crucially, fine-tuning demonstrated substantial recovery, with Cloe reaching 89.8% on SST-2 (95.6% of the matched teacher) and 79.4% teacher retention on XSum. The degradation is attributed to a combination of information loss during quantization and incomplete recovery due to the limited training budget. The research also highlighted an evaluation pitfall: standard answer-letter scoring failed for Cloe, necessitating continuation scoring. Ultimately, while ternary conversion isn't a general drop-in replacement, it proves valuable as a compact foundation for models tailored to specific tasks.