Pooled Benchmarks Mislead on Root-Cause Analysis Performance

Common practice in offline root-cause analysis (RCA) benchmarks is to rank methods using a single, pooled top-1 accuracy score across multiple subsystems. This paper audits this practice, demonstrating that such pooled leaderboards can be misleading. The analysis, conducted across three public RCA benchmark families and 11 subsystems, showed that the "winning" method based on pooled scores often underperforms on individual subsystems. The study found significant subsystem-level effects, with pairwise comparisons frequently showing different methods excelling in different contexts. Relying solely on pooled scores can lead to selecting a suboptimal method for a specific subsystem, resulting in substantial performance regret. The authors advocate for more granular, per-subsystem reporting protocols to provide a clearer and more actionable picture of method performance.