DumpsterCluster Serves LLaMA-70B on Used GPUs, Highlighting Sustainability Trade-offs

Researchers investigated the potential of using retired GPUs from secondary markets to form a "DumpsterCluster" capable of serving modern large language model (LLM) inference. They physically constructed a 128-GPU cluster using only second-hand components and operated it for a year, successfully serving LLaMA-70B. The economic benefits were substantial, with the DumpsterCluster costing significantly less than a new, high-end system. Through pipeline-parallel optimizations, the V100-based cluster achieved competitive LLaMA-70B throughput, demonstrating its production viability. However, the study also highlighted critical context dependencies and environmental trade-offs. Older GPUs consume considerably more energy per token, making the total cost of ownership favorable only in regions with inexpensive electricity. Furthermore, under grid-average carbon intensity, second-hand systems can produce substantially higher total carbon emissions per token, particularly for larger models like 70B. These findings suggest that while GPU repurposing can expand AI capacity affordably, it is not universally sustainable and must be strategically paired with low-carbon energy sources to be environmentally responsible.