Agentic AI for Safety-Critical Multi-Drone Systems

Multi-drone systems are increasingly being considered for safety-critical applications such as search and rescue or infrastructure monitoring. However, their widespread adoption is hindered not just by autonomy performance, but by the complex challenge of integrating agentic AI behaviors into professional human workflows. Operators must be able to understand, trust, and effectively govern these automated systems, especially under pressure and with accountability. This paper synthesizes insights from two ongoing projects, NAMUR and PERSIST, which explore LLM-supported robot control in emergency contexts and persistent drone operations for security. The authors argue that agentic AI in these sensitive domains should be approached as a socio-technical design problem. This means that user interfaces, oversight mechanisms, and evaluation practices are as crucial as the underlying algorithms themselves. The research advocates for a human-centered, participatory, and iterative design methodology. This approach aims to uncover stakeholder needs, progressively shape agent capabilities through prototypes, and ultimately produce transferable proof-of-concept systems and evaluation strategies suitable for other safety-critical environments. The focus is on ensuring that AI agents enhance, rather than complicate, human decision-making and operational safety.