Standardized ML Evaluation for Power System Protection

Researchers have introduced a standardized framework designed to bring consistency and comparability to the evaluation of machine learning models in power system protection. Currently, studies often report near-perfect scores, but these results are difficult to compare due to wide variations in evaluation settings, including protection tasks, physical scope, measurements, timing, and validation protocols. The proposed framework aims to make evaluation design an explicit part of the scientific contribution. The framework outlines seven essential study dimensions: protection objective, physical scope, observability, timing and decision windows, targets and sample validity, validation protocol, and evaluation outputs. To demonstrate its utility, the paper applies the framework to a case study using the PROTECT-90 benchmark for fault classification and localization. The study highlights how different evaluation assumptions, such as observability and decision horizons, significantly impact performance, emphasizing the need for explicit, reproducible evidence to enable more auditable and certifiable ML protection functions.