LLMs Evaluated for Design Structure Matrix Generation in Engineering.

This research introduces a novel black-box evaluation framework designed to rigorously test the capabilities of Large Language Models (LLMs) in automatically generating Design Structure Matrices (DSMs). DSMs are crucial tools in systems engineering for mapping dependencies between components. The framework compares LLM-generated DSMs against manually validated ground-truth matrices, employing a comprehensive set of metrics including structural completeness, correctness, coupling density, and stability measures. Experiments conducted on both synthetic and real-world datasets demonstrate that while LLMs can produce structurally sound DSMs, their performance is highly susceptible to the clarity and consistency of the input documentation and the specific phrasing of prompts. The study identifies common failure modes such as hallucinations and abstention, underscoring both the promise and current limitations of using LLMs for automated DSM generation. The proposed framework offers a transparent method for auditing Auto-DSM pipelines, laying the groundwork for integrating LLM-based decomposition techniques into model-based systems engineering (MBSE) workflows. This provides a standardized way to assess and improve the reliability of AI-driven design tools.