BrainBench Evaluates LLM Understanding of EEG Data

Analyzing Electroencephalography (EEG) data is a complex task that goes beyond merely assigning labels; it requires integrating natural language instructions, signal processing, quantitative evidence, and scientific interpretation. Current evaluation methods for Large Language Models (LLMs) primarily focus on isolated decoding tasks, failing to adequately quantify their ability for this comprehensive EEG understanding. To address this gap, a new benchmark called BrainBench has been developed. It provides a unified framework for evaluating LLMs on instruction-conditioned EEG understanding, featuring four subsets: Foundational Analysis, Sleep Assessment, Neurocognitive Assessment, and Physiological Integration. These subsets encompass 17 datasets and numerous tasks, using real-world data instances. Systems are required to perform analyses and generate scientifically grounded reports, potentially including artifacts. Initial evaluations of several LLMs under different execution paradigms (autonomous code execution with CodeAct and structured agentic analysis with BrainAgent) reveal significant performance variations, indicating that an LLM's competence in EEG understanding depends heavily on the model itself and its operationalization.