Learnijoy
ResearchAI Engineering & DevToolsAI Research

PHITSBench: AI Benchmark for Radiation Transport Modeling

Xianglin Ji, Svetlana V. Boriskina· July 14, 2026 View original

▶ The 2-minute explainer

Key takeaways

  • AI can assist with editing and repairing scientific simulation inputs effectively.
  • Domain-specific knowledge is crucial for AI to generate complex simulations from scratch.
  • Agentic execution further improves AI performance in scientific tasks.
  • Future AI progress in scientific modeling requires better knowledge bases and evaluation.

Who benefits

Nuclear EnergyAerospaceScientific ResearchDefenseHealthcare (radiation therapy)

Summary

PHITSBench is a new execution-scored benchmark for AI-assisted generation of PHITS radiation-transport inputs from natural language, featuring 282 tasks across editing, repair, and full simulation generation. It evaluates GPT-5.4 configurations, showing significant improvements with domain-specific knowledge and agentic execution.

A new benchmark, PHITSBench, has been introduced to evaluate AI's capability in generating inputs for the Monte Carlo Particle and Heavy Ion Transport code System (PHITS) using natural language. This benchmark comprises 282 tasks categorized into parameter editing, syntax repair, and complete simulation generation from descriptive text. Each task is scored based on execution success and the agreement between generated and reference transport observables. The study tested five configurations of GPT-5.4 models. Without specialized domain knowledge, the models performed well on editing (95% success) and repair tasks (70% success) but completely failed at generating full simulations from scratch (0% success). However, providing a structured, machine-readable PHITS knowledge catalog alongside the user manual dramatically improved single-shot simulation generation success to 57%. Further enhancements were observed with agentic execution, pushing success rates to 66-73%, albeit with increased computational cost. Failure analysis indicated that remaining errors primarily stemmed from incorrect selection and configuration of physical observables, rather than syntax issues. This suggests that future progress in AI-assisted radiation transport modeling will heavily rely on curated machine-readable knowledge bases, domain-specific training data, and execution-grounded evaluation environments, in addition to foundational model advancements.

Why it matters

For professionals in scientific computing, nuclear engineering, and related fields, automating complex simulation input generation can drastically improve efficiency and reduce errors. This benchmark highlights the current capabilities and critical needs for AI in specialized scientific domains.

How to implement this in your domain

  1. 1Develop machine-readable knowledge bases for domain-specific tools and simulation software.
  2. 2Curate domain-specific training datasets for fine-tuning large language models.
  3. 3Implement execution-grounded evaluation environments to validate AI-generated outputs.
  4. 4Explore agentic workflows to enhance AI's ability to handle complex, multi-step scientific tasks.
  5. 5Focus on improving AI's understanding and configuration of physical observables in scientific modeling.

Original post by Xianglin Ji, Svetlana V. Boriskina

"arXiv:2607.09789v1 Announce Type: new Abstract: We introduce PHITSBench, an execution-scored benchmark for the Monte Carlo Particle and Heavy Ion Transport code System (PHITS). PHITSBench comprises 282 transport-scorable tasks spanning three common workflow categories: parameter…"

View on X

Originally posted by Xianglin Ji, Svetlana V. Boriskina on X · view source

Want to go deeper?

Turn these trends into skills with Learnijoy's hands-on AI & tech courses.

Explore courses

More in AI Engineering & DevTools

AI Engineering & DevTools

Zapier vs. Tray: Enterprise Automation Platform Comparison for 2026

This post compares Zapier and Tray.io, evaluating which platform is better suited for enterprise automation needs by balancing power and ease of use. It argues that the best tools scale for complex requirements while remaining intuitive for all users.

Ryan KaneJul 28, 2027
AI InvestingAI Engineering & DevTools

Cross-Regime Bayesian Optimization Boosts Algorithmic Trading Signals

This paper introduces a cross-regime Bayesian optimization approach for hyperparameter selection in algorithmic trading, targeting robustness across different market regimes. It finds that a hybrid ensemble of XGBoost and TabNet achieves an annualized return of 51.26% and a Sharpe ratio of 2.44, outperforming individual models and demonstrating significant out-of-sample generalization.

Joshua Le GriceAug 28, 2026
AI ResearchAI Engineering & DevTools

Emotional Preferences Regulate Goal Priorities in Reinforcement Learning Agents

This paper proposes a computational framework where higher-level goals autonomously generate state-dependent emotional preferences to regulate the priorities of competing lower-level objectives in reinforcement learning agents. It demonstrates how this emergent preference function exhibits contextual priority switching and improves performance over fixed-preference strategies in multi-objective exploration environments.

Shiqi Liu, Yihua Tan, Hu Fu, Guanyu QiAug 28, 2026