Distillation Recipe for CPU-Deployable LLM Safety Classifiers

Deploying safety layers for Large Language Models (LLMs) on standard CPU hardware faces significant challenges due to the large size of current open guard models, which are typically optimized for GPUs and incur slow inference times on CPUs. This research introduces a reproducible and license-aware knowledge distillation recipe designed to overcome these constraints, enabling the creation of smaller, CPU-deployable safety classifiers. The process involves using a powerful open guard model to label a large corpus of approximately 97,000 prompts, drawn from 24 public datasets, into seven safety categories based on a public hazard taxonomy. A fleet of smaller "student" models, encompassing lexical, shallow, encoder, and generative architectures, are then trained to replicate this safety signal. A key aspect is the partitioning of the training corpus by license, allowing for a measurable comparison between deployable and research models based solely on data restrictions. Evaluated against an independent gold benchmark of 6,361 rows, the distilled student models demonstrate performance comparable to their larger teachers on adversarial text, within overlapping confidence intervals. Crucially, these smaller models also reduce false alarms on harmless prompts; the smallest generative student achieved a 3.8% false alarm rate compared to 4.8% for the 8-billion-parameter teacher. The encoder model, for instance, classifies requests in roughly 24 milliseconds on a CPU. The study highlights that per-class rebalancing is the most critical ingredient for the recipe's success.