LLMs Benchmark for mmWave Radar Human Perception

Large language models (LLMs) have demonstrated impressive reasoning and generative capabilities, leading to their consideration as universal reasoning engines for various perception tasks. While vision-language models (VLMs) have integrated LLMs with visual sensing, their application to millimeter-wave (mmWave) radar data, which offers unique advantages in low-light and occluded conditions, remains largely unexplored. Key obstacles include a lack of radar-language paired data, significant cross-dataset heterogeneity, and the absence of a foundational mmWave encoder. To bridge this gap, researchers developed a minimal textualization interface that converts mmWave point clouds into concise natural language descriptions. This allows off-the-shelf LLMs to process mmWave data within a question-answering (QA) framework. Building on this, they present mmWave-QA, the first benchmark specifically designed for language-conditioned mmWave human perception. The mmWave-QA benchmark aggregates diverse public mmWave datasets, harmonizing them through calibration-aware preprocessing and global taxonomy alignment, and provides natural language QA pairs. Covering six scenarios and five QA tasks, it enables standardized evaluation across different mmWave hardware and experimental conditions. Initial evaluations of LLMs on mmWave-QA reveal their promising zero-shot reasoning potential for radar perception and demonstrate their robustness even under visual degradation.