LLM-as-Judge Safety Evaluations Lack Reproducibility, Even at Zero Temperature

This research uncovers significant reproducibility issues in LLM-as-judge safety evaluations, a standard component in AI deployment decision-making. A common assumption is that setting the sampling temperature to 0 ensures deterministic grading. However, testing against a real-world safety evaluation codebase (Japan AISI's aisev) demonstrated this assumption is flawed on two levels. Firstly, many harnesses invoke graders without explicitly setting temperature or seed, leading providers to silently apply a default of 1.0. This results in substantial per-item disagreement (up to 50%) across identical runs for items near decision boundaries. Secondly, even when temperature is explicitly pinned to 0, non-reproducibility persists. Across 690 API calls with various providers and models, 1-2 out of 7 borderline items remained non-reproducible, even under forced greedy decoding. The deprecation of temperature control in newer models like Claude Opus 4.7/4.8 further complicates mitigation. These findings highlight a structural gap where single-run verdicts without variance metrics can mask noise as a safety property.