UTP-Bench Evaluates LLM Travel Planning Under Real-World Uncertainty

Large Language Models (LLMs) have recently shown impressive capabilities in generating automated travel itineraries. However, real-world travel planning is inherently unpredictable, with frequent disruptions like transportation delays, fluctuating crowd densities, and unexpected stochastic delays often invalidating otherwise sound schedules. Existing benchmarks, such as TravelPlanner and TripCraft, operate under deterministic assumptions, evaluating only static constraint satisfaction and failing to assess how robust generated plans remain when uncertainties arise. To address this critical limitation, researchers introduce UTP-Bench, a comprehensive benchmark for uncertainty-aware travel planning. UTP-Bench is a large-scale dataset that incorporates real-world travel information from 504 Indian cities, including attractions, restaurants, accommodations, and multi-modal transportation networks. To accurately model realistic disruptions, the benchmark integrates empirical delay distributions and crowd-density patterns collected from major cities, enabling the evaluation of travel plans under stochastic conditions. The researchers also propose three novel evaluation metrics: Buffer Adequacy Score (BAS), Crowd-Aware Timing Score (CATS), and Transport Delay Absorption Score (TDAS). These metrics quantify an itinerary's ability to maintain robustness against transit delays and crowd variability. Experiments conducted with state-of-the-art LLMs like GPT-5, Qwen3, Mistral, and Phi-4 reveal substantial discrepancies between model-generated and human-authored plans, particularly concerning temporal buffering, delay-aware transportation scheduling, and crowd-sensitive planning.