Best Metrics for ERP-Based BCI Spelling Rate Accuracy

In Brain-Computer Interface (BCI) systems, particularly those based on Event-Related Potentials (ERPs), the spelling rate—the number of correctly selected characters—is a more critical performance indicator than traditional loss or accuracy metrics. This is because spelling rate directly influences the information transfer rate (ITR) and overall spelling performance. Furthermore, ERP-based BCIs typically involve imbalanced data class distributions, necessitating the use of metrics that can robustly handle such imbalances, like the area under the receiver operating characteristic curve (ROC AUC). This study systematically investigates the correlation between the spelling rate and 13 different performance metrics to determine which ones best reflect user spelling performance and how they are affected by trial repetition. The research utilized two distinct datasets: a private LARESI ERP dataset and the public OpenBMI ERP dataset, ensuring a broad evaluation. The findings strongly suggest that the Brier score, Matthews Correlation Coefficient (MCC), and several metrics designed for imbalanced binary classification—specifically ROC AUC, area under the Precision-Recall curve (PR AUC), Average Precision (AP), and partial AUC (pAUC)—are the most reliable indicators. These metrics are encouraged for reporting in future ERP-based BCI experiments, providing a more accurate and comprehensive assessment of system performance.