New Tool Reveals Training Data Leakage in Black-Box LLMs

Researchers have developed a new probabilistic method, dubbed "Leak It," to analyze and extract training data from black-box language models, shedding light on privacy vulnerabilities often obscured by aggregate metrics. Traditional membership inference attacks (MIA) using ROC-AUC can be misleading, as simple blind baselines can often distinguish between members and non-members based on surface text alone, without needing model interaction. The study extends this critique to the sampling regime, demonstrating that while aggregate metrics may not show significant incremental gains from sampling, the method can verbatim-extract specific training data for a tail of documents that blind attacks cannot reach. For instance, on Pythia-6.9B, a significant percentage of documents containing real identifiers (like email addresses) had those identifiers reproduced and uniquely attributable to the document. This per-document leakage risk is uneven, being approximately three times stronger in code than in prose, and it increases with model capacity. Crucially, factors like temperature, nucleus sampling, and corpus deduplication showed little to no reduction in this leakage. The research advocates for privacy audits to report per-document extraction, broken down by domain, rather than relying solely on single aggregate AUC scores.