LLMs Generate Simulation Code for Fluid Systems: Benchmarking Performance

Researchers investigated the capability of large language models (LLMs) to automate the generation of simulation code for complex fluid systems. The goal was to convert a neutral graph representation of these systems into functional code compatible with established simulation environments like the Python library WNTR and the Modelica Standard Library. The study conducted a comprehensive comparison, evaluating ten different state-of-the-art LLMs alongside six distinct prompting strategies. These strategies varied in the amount and type of contextual information provided to the models, such as code examples or documentation snippets. The generated code was then assessed based on standard software quality metrics, and the functional accuracy of the resulting simulation models was validated by replicating benchmark fluid system scenarios. While the top-performing LLM configurations achieved satisfactory syntactic code quality, the research highlighted significant remaining challenges in ensuring high simulation fidelity. The findings offer practical guidance for engineers and researchers looking to integrate LLM-driven code synthesis into their model-based design workflows, emphasizing areas where further improvement is needed.