WARP Recovers Foundation Model Training Data Portfolios from Weights

The training data recipes for large foundation models, particularly the domain mixture weights that dictate how different data sources are sampled, are rarely disclosed to the public. This lack of transparency creates a significant challenge for researchers attempting to understand and analyze these models. Traditional methods, such as membership inference, only identify individual training samples and cannot characterize the overall composition of the training corpus. To address this, a new framework called WARP (Weight-Space Analysis for Recovering Training Data Portfolios) has been introduced. WARP works by analyzing the released weights of a fine-tuned model to infer its training data mixtures. It achieves this by interpolating between the base model and the fine-tuned model using model merging techniques, which generates "pseudo-checkpoints" that simulate the training trajectory. This process reveals a geometric footprint of the training data within the model's weight space. From these simulated footprints, WARP extracts geometric features and then maps them to domain proportions. This mapping can be done either through a parameter-free softmax readout or by using an MLP projector trained on synthetic mixtures. In controlled experiments with models like BERT and GPT-2, WARP demonstrated impressive accuracy, recovering domain mixtures with average Mean Absolute Error (MAE) as low as 0.046 and 0.104, respectively. This performance significantly surpasses that of membership inference and even a variant with access to the true training trajectory, offering a powerful tool for understanding model provenance.