CARGO: Training-Free LLM Offloading for Local-Cloud Collaboration

Deploying large language models (LLMs) under resource constraints often involves local-cloud collaboration, where a smaller local model handles simpler tasks and offloads complex ones to a more powerful cloud model. Existing methods typically rely on trained routers or fine-tuning, which tie routing behavior to specific operating conditions. This research proposes that such training might be unnecessary. The new framework, CARGO (Controllable-Ratio LLM Offloading via Reliability Gating), demonstrates that a local model's own inference-time agreement across sampled responses provides a strong indicator for deciding when to trust local execution versus offloading to a cloud model. CARGO estimates this agreement through prompt-varied sampling, uses Bayesian early stopping for efficient uncertainty control, and allows for arbitrary target collaboration ratios via simple deployment-time calibration. Across diverse tasks and LLM families, CARGO consistently outperforms other training-free baselines and, in some cases, even surpasses supervised learned routers, suggesting that effective local-cloud collaboration can emerge directly from the local model's intrinsic behavior.