FactoryLLM: Safe Open-Source AI Playground for Smart Factories

Diagnosing and recovering from faults in smart factories presents a significant challenge due to the vast amount of critical information dispersed across numerous machine manuals and interconnected manufacturing processes. Large Language Models (LLMs) offer a promising solution for navigating this complexity. This paper introduces FactoryLLM, an open-source and secure AI testing environment specifically developed for evaluating different LLM-based RAG models. It allows users to analyze documentation from multiple machines across an entire manufacturing process. A key feature is its safety, as it supports running local or open-source LLMs, ensuring that sensitive industrial data remains within the user's control and is not shared externally. FactoryLLM provides a dual evaluation setup, utilizing both RAGAS and NVIDIA's LLM-as-a-Judge metrics, to assess model performance in multi-document reasoning. A case study involving an Autonomous Intelligent Vehicle demonstrated its effectiveness, with models achieving high groundedness scores in cross-machine document reasoning. The full code and documentation are publicly available for community use.