Anomaly Detection Algorithm Rankings Unreliable Due to Benchmarking Inconsistencies

Research indicates that the perceived performance hierarchy of anomaly detection algorithms is often misleading. The study found that depending on how benchmarks are constructed—specifically, the datasets chosen and the hyperparameters configured—nearly any leading algorithm can be made to appear superior. This variability casts doubt on the reliability and reproducibility of current state-of-the-art claims in the field. The instability metric introduced in the work quantifies this effect, showing that common benchmarking practices lead to significant fluctuations in algorithm rankings. The findings suggest that current benchmark suites are often insufficient, requiring larger and more diverse dataset collections to provide stable and meaningful comparisons. While random seeds and evaluation metrics have a lesser impact, the selection of data and tuning parameters are critical factors that can drastically alter an algorithm's perceived effectiveness. This calls for a re-evaluation of how anomaly detection methods are assessed and compared.