New Benchmark Reveals Document Sensitivity Classification Challenges

Automatically classifying the sensitivity of organizational documents is a critical task, with misclassifications potentially leading to severe regulatory or security issues. While AI offers a scalable solution, its reliability hinges on the quality of training data. A significant, often overlooked problem is "label leakage," where models exploit superficial markers in documents rather than learning genuine content-based sensitivity, leading to inflated performance metrics. To address this, researchers have developed Strategic 16K, a meticulously curated dataset of 16,000 diplomatic cables from the WikiLeaks Public Library of US Diplomacy (PlusD). This dataset is specifically designed to be leakage-controlled, meaning it has undergone an extensive protocol to remove residual classification markers that could mislead models. The study benchmarks six different model architectures, including classical machine learning methods and transformer-based approaches, on this clean dataset. BERT emerged as the top performer, followed closely by ELECTRA. Among classical models, TF-IDF with Logistic Regression showed strong performance at a much lower computational cost. This work provides the first fully reproducible, leakage-controlled benchmark for document sensitivity classification using WikiLeaks PlusD data.