Agora Enables Collective, Permissionless Internet-Scale LLM Pretraining

The training of large language models (LLMs) at multi-billion to trillion parameter scales is currently restricted to specialized datacenters, which rely on homogeneous accelerators, high-speed interconnects, and centralized orchestration. This concentration limits frontier model development to a select few organizations. Meanwhile, a vast, untapped pool of compute power exists in individually owned, heterogeneous, and preemptible consumer and professional GPUs connected only by the internet. Agora is a groundbreaking system designed to harness this distributed compute. It achieves this by combining bandwidth-efficient pipeline-parallel model sharding, optimized for internet-grade links, with multi-party, fault-tolerant collective operations. A key aspect of Agora is "Protocol Learning," where each participant holds only one stage of the model, and no single party ever possesses the complete model weights. This setup enables collectively trained and owned models, paving the way for economically sustainable, open-source frontier model development. This report details the culmination of extensive research in communication-efficient parallelism, asynchronous optimization, and fault-tolerant systems. It presents the first-of-its-kind demonstration: Pluralis-8B, an 8.6-billion-parameter model pretrained permissionlessly on 500 billion tokens of FineWeb-Edu. This training run spanned 40 days, involved 330 contributor nodes (predominantly consumer GPUs), and sustained approximately 170,000 tokens per second. The system achieved 63% of the efficiency of a centralized H100 baseline and converged to within a small margin of a centralized reference run, even with participants joining and leaving throughout the process.