Auditing Reveals Flaws in AI Selective Prediction Risk Control

Selective prediction systems, which only make predictions on inputs they are confident about, often promise a controlled error rate for accepted inputs. This promise is typically backed by statistical guarantees. However, a recent audit of these systems, particularly in signal classification domains like anomalous sound detection and AI-generated image forensics, reveals significant vulnerabilities. The audit found that a commonly used, uncertified empirical thresholding method frequently fails to meet its declared error budget, sometimes exceeding it in up to 73% of trials. This creates a dangerous "false sense of safety." While certified methods like Clopper-Pearson and betting bounds offer tighter guarantees and perform well under ideal conditions, their reliability collapses when the underlying data distribution changes, such as when encountering unseen machine types or image generators. Under these "grouped deployment" scenarios, even certified rules overran their budgets in a substantial percentage of trials, indicating a failure of the exchangeability premise rather than the bounds themselves. This suggests that while statistical bounds are mathematically sound, their practical application requires careful consideration of data distribution shifts to avoid misleading safety assurances.