New Benchmark Evaluates LLM Editing Capabilities for Building Information Models.

The application of Large Language Models (LLMs) in computer-aided design (CAD) is growing, particularly for generating designs from text. However, real-world engineering requires LLMs to not only create new geometry but also to understand, correctly edit, and preserve the semantics and relationships within existing models. Many current CAD benchmarks primarily focus on new model creation and geometric accuracy, overlooking the complexities of editing. To address this gap, a new benchmark called BIM-Edit has been introduced. It specifically evaluates LLMs on natural-language editing tasks within Building Information Models (BIM), which are represented in the Industry Foundation Classes (IFC) format. BIM provides a challenging environment due to its intricate encoding of geometry alongside semantic and relational structures. BIM-Edit comprises 324 editing tasks across 11 realistic and 36 synthetic building models, categorized into direct, spatial, and topological instructions. The evaluation considers geometric accuracy, semantic validity, and topological consistency. Results indicate that even the best-performing LLMs achieve only a 49.5% average score, with very few tasks fully solved, revealing a substantial deficiency in current LLM capabilities for structured engineering design workflows.