ProbGuard Estimates LLM Safety Risk from Output Distributions

Current safety guardrails for Large Language Models (LLMs) typically treat safety assessment as a deterministic classification task, converting a token sequence into a discrete safety label. This approach has two key limitations: it ignores the inherent uncertainty in safety assessment, especially during early generation, and it discards the rich probabilistic information embedded in the LLM's output distribution. To address these shortcomings, researchers propose ProbGuard, the first fully probabilistic and architecture-agnostic guardrail. ProbGuard leverages the early output distributional signals from an LLM to estimate and calibrate the safety probability of its ongoing generation. It formulates safety risk as the unsafe probability of continued generation dynamics, estimated through Monte-Carlo sampling. By post-training on these distributional signals and calibrated safety risk, ProbGuard achieves superior calibration performance across various model-dataset combinations, reducing average Brier score and ECE significantly. Furthermore, ProbGuard effectively limits attack success rates to at most 1% across six jailbreak attacks, even when observing only the first ten decoding steps of the LLM's output. This capability for early stopping of unsafe generations marks a significant advancement in LLM safety.