New Method Detects Unsafe LLM Outputs Using Embedding Dynamics

The increasing deployment of Large Language Models (LLMs) in critical applications highlights the urgent need for robust methods to detect and prevent the generation of harmful, toxic, or policy-violating content. Current detection methods often struggle with efficiency and black-box applicability. Researchers propose an extension of a dynamical systems framework, previously used for hallucination detection, to classify LLM safety. This novel approach projects both the prompt and the LLM's response into high-dimensional embedding spaces. Separate Koopman-based predictive models are then fitted for what constitutes "safe" and "unsafe" content regimes. Safety classification is achieved by comparing the prediction errors of these safe and unsafe models using a new differential residual score. A key innovation is the incorporation of prompt and response embedding dynamics, allowing the fitted Koopman operators to capture crucial interaction patterns. Evaluations across three safety benchmarks and three embedding models show that including prompt embeddings consistently improves detection, especially for violations dependent on prompt-response interaction, while response-only violations benefit from dense semantic embeddings. This work opens new avenues for analyzing AI systems using dynamical systems principles.