AFDBench: AI Meteorologist Generates Accurate Weather Forecast Discussions

Large language models (LLMs) often hallucinate numerical values when generating high-stakes meteorological text, posing significant risks in weather communication. To address this, AFDBench has been introduced as an AI meteorologist capable of generating professional Area Forecast Discussions (AFDs) by reasoning through structured AI weather forecast data from Google's WeatherNext 2. AFDBench also serves as the first benchmark specifically designed for evaluating generative meteorological reasoning. It comprises 7,732 expert-written discussions from 13 National Weather Service (NWS) offices, paired with real AI weather forecast inputs. The benchmark uses three metrics: Met-Align for numerical accuracy, Style-Align for adherence to professional dialect, and Input-Grounding for fidelity to source data. Zero-shot evaluations revealed that open-source LLMs performed poorly on Style-Align and moderately on Input-Grounding. However, applying Group Relative Policy Optimization (GRPO) with domain-specific rewards targeting temperature accuracy, synoptic correctness, and format compliance dramatically improved performance. On unseen NWS offices, GRPO nearly doubled Style-Align from 0.318 to 0.619 and improved Input-Grounding from 0.881 to 0.940, demonstrating that reinforcement learning can teach a 7B-parameter model to write like a professional meteorologist and accurately interpret AI weather data.