Automated Safety Benchmarks Insufficient for Small Language Models

As Small Language Models (SLMs) become more prevalent in resource-constrained and privacy-sensitive environments, ensuring their safety and mitigating biases is paramount. However, current automated AI safety benchmarks, primarily designed for larger language models, may not reliably transfer to SLMs. This research investigates the effectiveness and robustness of these benchmarks when applied to SLMs. The study conducted a comprehensive assessment across five widely used benchmark suites and 26 open-source SLMs, using a unified judging rubric that scores responses as harmful, safe, or ambiguous/irrelevant. A key finding was the prevalence of ambiguous judgments, which correlated with factors like prompt complexity and model architecture. This suggests that benchmarks designed for LLMs are inadequate as standalone evidence for SLM safety. Furthermore, the research identified a "capability-safety confound," where ambiguity rates increased with lexical density, output perplexity, and length, while decreasing with lexical sophistication and self-coherence. This means that apparent safety can be intertwined with a model's general capability. The high ambiguity rate also renders aggregate mean-score leaderboards mathematically brittle, as model rankings can change significantly with different treatments of ambiguous results, even if the underlying outputs remain the same.