Federated Pre-Training Evaluation: Downstream Fine-Tuning Can Be Misleading

Federated pre-training allows foundation models to be trained on decentralized, private datasets without centralizing the data. However, assessing the quality of these models has proven difficult, particularly because variations in client participation and local data availability complicate direct comparisons. Current evaluation methods often rely on downstream fine-tuning, which involves adapting the pre-trained model to specific tasks. This study investigated the reliability of different evaluation protocols for federated pre-training. Researchers compared downstream fine-tuning on benchmarks like GLUE with intrinsic evaluation methods, specifically next-token prediction. They found that downstream fine-tuning did not consistently preserve the model ranking established during pre-training, suggesting it can be an unreliable indicator of true model quality. In contrast, direct next-token prediction showed a strong correlation with the pre-training test perplexity, indicating it more accurately reflects the model's foundational learning. The findings highlight that relying solely on downstream fine-tuning for federated pre-trained models can be misleading, and intrinsic evaluation methods should receive greater attention.