New Data Poisoning Attack Manipulates AI World Models Stealthily.

This research unveils a significant security vulnerability in AI systems that rely on learned world models for prediction, planning, and adaptation. The paper introduces SWAAP, a novel two-stage data poisoning framework designed to stealthily manipulate these world models during their fine-tuning phase. This manipulation can corrupt the learned dynamics, leading to flawed downstream planning and suboptimal agent behavior. SWAAP operates by first identifying a malicious target world model that, despite appearing similar to clean dynamics, causes agents to exhibit low-return actions. This is achieved using a sophisticated optimization technique. In the second stage, SWAAP subtly modifies a small portion of fine-tuning data, ensuring that the resulting training gradients guide the victim model towards the adversarial target. Crucially, these poisoned data points are designed to remain close to the model's natural prediction errors, enhancing the attack's stealth. The effectiveness and stealth of SWAAP were rigorously tested against various defenses, including pre-training detection, robust fine-tuning, and test-time monitoring. Across multiple continuous-control tasks, SWAAP consistently induced substantial performance degradation in AI agents while successfully evading several non-adaptive detection methods. These findings underscore a practical and concerning vulnerability in current world-model adaptation pipelines, emphasizing the urgent need for more robust protection mechanisms for both training data and learned model dynamics.