New Benchmark Evaluates LLM Handling of Off-Procedure Diagnostic Queries

Language models are increasingly deployed as advisory tools in maintenance and operational settings. To ensure accuracy and prevent fabricated responses, these systems are often "grounded" in specific procedural documentation, limiting their responses to approved steps. However, real-world operators frequently ask questions that fall outside these predefined procedures, a scenario not well-addressed by existing evaluation benchmarks. To tackle this, a new dataset called DiagFlowBench has been developed. It comprises 50 industrial diagnostic flowcharts from a consumer manufacturer, translated into 1,676 multi-turn conversations. These conversations specifically include both compliant and out-of-scope utterances, allowing for a direct assessment of how models handle deviations from protocol. Testing ten commercial and open-source language models on DiagFlowBench revealed significant variations in their ability to abstain from answering out-of-scope questions. A common issue observed was models selecting a real but contextually inappropriate step rather than explicitly indicating an inability to answer or fabricating information. This tendency to offer plausible but incorrect advice presents a subtle yet challenging vulnerability for systems relying on grounding.