AI Benchmark Audits Prone to Five Failure Modes

AI providers and auditors are increasingly required to provide documented evaluation evidence, often in the form of perturbation-based construct-validity audits. However, new research suggests these audits are inherently fragile, with their conclusions susceptible to unseen implementation details. The paper outlines five distinct failure modes within the audit pipeline, illustrating each with a self-audit on safety benchmarks using open-weight instruction-tuned models. The study applied a unified six-point due-diligence gate to its findings, revealing that none of the audited cells reached a confirmatory status. This framework is presented as a protocol for withholding and disclosing assurance-grade evidence, intended to supplement existing construct-validity methods rather than replace them. The identified failure modes are an initial taxonomy, not an exhaustive list, and the evidence is based on a specific case study.