On-Premise AI Agents Optimized for Factory Hardware

Deploying conversational AI assistants on factory shop floors to provide workers with access to machine documentation presents a significant challenge: powerful models often exceed the capabilities of on-premise hardware. This paper introduces a novel approach to address this by focusing on sub-network selection after model compression and retrieval-grounded adaptation. The study reveals a counter-intuitive finding: once a model undergoes structural compression and is adapted for retrieval-augmented generation, its size (parameter count) no longer reliably predicts the quality of its answers. While general capabilities tend to decrease linearly with parameter reduction, the quality of answers when augmented with retrieval does not follow the same pattern. This suggests that optimizing for size alone is insufficient. Consequently, the authors propose treating deployment as a post-adaptation selection problem. For each device, a specific sub-network is chosen based on its judged answer quality and measured on-device throughput, while adhering to a configurable general-capability floor and memory budget. Strategies that solely optimize for size, speed, or quality individually lead to compromises in other areas. To make this selection process cost-effective, a weight-shared supernetwork is trained using sandwich-style in-place distillation. A case study involving manufacturing manuals demonstrated that initial quality loss from extraction was largely recovered after retrieval-grounded distillation, and the same assistant could run efficiently across various edge tiers with low power consumption.