LLMs Show Promising, Yet Flawed, Mechanical Engineering Awareness

While Large Language Models (LLMs) excel in code generation and mathematical reasoning, their proficiency in mechanical engineering, specifically mechanics and spatial geometry, has been less systematically quantified. A new automated benchmark called MecEng addresses this by evaluating LLMs on their capacity to create multibody simulation models from parameterized textual descriptions. The benchmark includes 84 tasks of varying difficulty, from simple rigid-body systems to complex flexible multibody systems requiring 3D geometry generation and finite-element meshing. The evaluation pipeline uses LLMs to generate simulation-ready geometry and build models for the Exudyn code, which are then verified against expert ground truth. The study assessed 32 open-weight and two proprietary LLMs. Results indicate that the best open-weight model achieved an 86.0% success rate on rigid-body tasks, close to the strongest proprietary model's 91.4%. However, flexible multibody tasks proved considerably more difficult for all models. The findings suggest that while LLMs are rapidly improving in mechanical engineering awareness, they remain prone to errors, particularly in more complex scenarios involving deformable bodies.