LLM-Guided Planning Boosts Nuclear Document Reasoning Accuracy

Researchers have developed an innovative AI agent designed to navigate and reason through extensive nuclear regulatory documents, a task traditionally requiring significant human effort due to its multi-hop nature. This agent employs an LLM to guide its planning, deciding which document fragments to examine next and when sufficient evidence has been gathered. It operates without traditional vector indexing, instead using browse, read, and search tools over a document tree and building a dynamic knowledge graph to track its progress and findings. The system demonstrated strong performance on a benchmark of 200 questions related to NuScale's Final Safety Analysis Report. It achieved an 81.5% accuracy rate, showing a substantial improvement over other RAG-based approaches, with planning identified as the key factor in its success. The approach also includes a traceability module for auditing, which can flag potential scope boundaries or partial conformance issues.