New Benchmark Evaluates LLMs for Aviation Operations

Large language models (LLMs) are increasingly being considered for various aviation business operations, including documentation, training, and customer support. However, existing general-purpose benchmarks fail to adequately assess an LLM's ability to reason safely and correctly about specialized aviation operational knowledge, a critical gap given the high-stakes and regulated nature of the industry. To address this, researchers have introduced Pre-Flight, an open-source benchmark comprising 300 multiple-choice questions. These questions are derived from international standards, airport ground operations materials, ICAO and US FAA regulations, and complex operational scenarios, authored and reviewed by aviation practitioners. The benchmark evaluates a range of commercial and open-weight models using the Inspect evaluation framework, scoring them by accuracy. While an informal expert reference score is around 95%, the strongest LLM evaluated (released in 2026) only achieved 82.7%, showing only gradual improvement from earlier models. This persistent and substantial gap below expert-level reliability underscores that LLMs are not yet ready for uncritical deployment in aviation. The dataset, evaluation harness, and results are publicly available, emphasizing that such domain-specific evaluation is a necessary prerequisite for the responsible integration of generative AI into non-safety-critical aviation operations.