New Semantic Overlays Technique Mitigates Prompt Injection Attacks

A new research paper introduces "Semantic Overlays," an innovative method designed to combat prompt injection attacks in large language models (LLMs). The fundamental problem addressed is that LLMs perceive all input as undifferentiated tokens, making it difficult for them to distinguish between user input, system instructions, or tool outputs, which attackers exploit to inject malicious commands. Semantic Overlays tackle this by introducing a non-textual communication channel directly into the model's input. These are small, learned adapters applied to a frozen model's residual stream at specific prefill positions. Unlike traditional steering vectors, overlays are trained, adaptable, and can be selectively applied to encode complex semantics, effectively creating an "out-of-band" annotation that tokens cannot replicate. The technique demonstrates strong results in mitigating prompt injection. For instance, an overlay marking a span as "non-executable" effectively defends against attacks that attempt to insert instructions into untrusted contexts. Experiments show significant improvements in separation benchmarks and drastic reductions in attack success rates across various prompt injection scenarios, all while maintaining the readability of the marked content.