Evaluating LLMs in Emergency Contexts: A Call to Action

The paper serves as a direct appeal to the research community, urging them to take a more active role in clearly articulating their findings to the public. It emphasizes the critical importance of transparent communication, especially concerning the capabilities and limitations of advanced AI technologies. To illustrate the high stakes involved, the authors present a case study focusing on the initial deployment of an LLM-based machine translation application. This system was designed for a text-to-911 service, advertised to support 55 languages for use in emergencies where direct calls might be difficult. The analysis reveals several prevalent misconceptions about such technologies. The paper concludes with a set of concrete recommendations and best practices for all stakeholders involved in the development and deployment pipeline. It argues that while scientific advancement often focuses on solving complex "hard" problems, it is frequently the "easy" problems—those for which cutting-edge technology might not even be necessary—that are most frequently overlooked, leading to significant real-world risks.