Language Models Leak Sensitive Data from Context Window.

A new study highlights a critical vulnerability in large language models (LLMs): the inadvertent leakage of sensitive user information. Even when LLMs are designed to refuse direct requests for private data, the mere presence of secrets like calendars, credentials, or health records within their context window can create subtle correlations in their benign outputs. This research demonstrates that these hidden correlations can be exploited by an adversary. Using a novel adaptive attack with black-box access to the model, secrets can be reconstructed. Experiments across eight proprietary models showed near-perfect accuracy for 2-digit secrets and 82% for 4-digit secrets, extracted from ordinary, non-adversarial requests. The findings suggest that more capable models are more prone to this leakage, indicating it might be an inherent byproduct of advanced instruction-following rather than a simple bug. Practical attacks include classifiers inferring semantic predicates about user memories and an RL-trained adversary extracting full Social Security Numbers from a production-style agent.