PHITSBench: AI Benchmark for Radiation Transport Modeling

A new benchmark, PHITSBench, has been introduced to evaluate AI's capability in generating inputs for the Monte Carlo Particle and Heavy Ion Transport code System (PHITS) using natural language. This benchmark comprises 282 tasks categorized into parameter editing, syntax repair, and complete simulation generation from descriptive text. Each task is scored based on execution success and the agreement between generated and reference transport observables. The study tested five configurations of GPT-5.4 models. Without specialized domain knowledge, the models performed well on editing (95% success) and repair tasks (70% success) but completely failed at generating full simulations from scratch (0% success). However, providing a structured, machine-readable PHITS knowledge catalog alongside the user manual dramatically improved single-shot simulation generation success to 57%. Further enhancements were observed with agentic execution, pushing success rates to 66-73%, albeit with increased computational cost. Failure analysis indicated that remaining errors primarily stemmed from incorrect selection and configuration of physical observables, rather than syntax issues. This suggests that future progress in AI-assisted radiation transport modeling will heavily rely on curated machine-readable knowledge bases, domain-specific training data, and execution-grounded evaluation environments, in addition to foundational model advancements.