Audit Reveals Flaws in Defensive Driving Evaluation Benchmarks

Defensive driving scores are intended to differentiate between various autonomous driving policies, particularly those that observe surrounding actors versus those that do not. However, a recent audit of the NAVSIM v2.2 evaluation system uncovered a significant vulnerability in its scoring methodology. The issue stems from "reference-conditioned forgiveness," a rule where an agent receives credit if the logged human reference also fails a compliance channel. When the agent and the reference share an unstable rollout transformation, this rule can inadvertently propagate shared reference failures, leading to inflated compliance credit for the agent. The audit demonstrated that under specific conditions, "route-blind" and "actor-blind" probes surprisingly outranked human replay and other sophisticated policies. The problem was traced to dependency-sensitive numerical behavior in the shared velocity refit, which caused rollout divergence. Replacing only the solver eliminated this divergence and restored expected policy rankings, underscoring the critical need for robust audit protocols including score basis disclosure, blind probes, and rollout stability tests in defensive driving benchmarks.