Auditing Autonomous AI Analysis Agents for Errors.

As autonomous agents increasingly perform entire data analyses—from cohort selection to model fitting—the challenge of identifying the source of errors when an analysis goes wrong becomes critical. This research rigorously examines "innovation-residual auditing," a method that learns from sound analyses to flag operations that deviate from predicted patterns, without requiring labeled mistakes.The study clarifies how the choice of scoring mechanism impacts error localization. Scoring operations based on deviation from the immediately preceding step can lead to a single error producing a single flag, while comparing against a longer reconstructed intended analysis spreads a single mistake across multiple operations. The paper quantifies this spread and advises on optimal comparison lengths for gradual error accumulation.Furthermore, the research introduces procedures to control the proportion of falsely flagged operations within an audited analysis, requiring only that sound analyses be exchangeable. It also quantifies how guarantees weaken with imperfect models or content-dependent review selection. A fundamental limit is established: errors below a certain magnitude are indistinguishable from normal variation and cannot be attributed, a limit primarily constrained by the representation's dimension rather than the volume of training data.