EMRB Benchmarks LLM Reasoning on Raw Electromagnetic Signals.

While Large Language Models (LLMs) are increasingly used as code agents for scientific and engineering analysis, their capability to process and interpret raw physical-layer measurements, such as electromagnetic signals, has remained largely untested. The "EMRB" (Electromagnetic Reasoning Benchmark) addresses this gap by providing 200 problems across five difficulty levels and 27 question types, ranging from signal detection to OFDM design. Unlike benchmarks that rely on preprocessed features, EMRB presents only raw I/Q data, requiring LLMs to discover relevant quantities through code generation and execution. The evaluation of 14 diverse LLMs on EMRB showed a wide range of scores, from 24.1% to 78.9%, with a sharp decline in performance on more complex tasks like system design compared to basic measurements. To enhance LLM performance, the researchers propose "ReconPilot," a structured method that separates signal reconnaissance, targeted analysis, and self-verification steps. ReconPilot significantly improved overall scores across various LLM backbones, demonstrating its effectiveness in boosting reasoning capabilities for raw signal analysis.