Pro-Router Boosts MLLM Inference Efficiency with Adaptive Edge-Cloud Routing

A new method called Pro-Router has been developed to enhance the efficiency of multimodal large language model (MLLM) inference, addressing the substantial computational overhead typically associated with these powerful models. Existing routing strategies often make decisions based on coarse request features or require multiple language model passes, overlooking valuable token-level uncertainty signals during generation. Pro-Router tackles this by implementing a two-stage progressive decision process. The first stage involves a lightweight prompt pre-scorer that quickly screens requests before token generation, directing simpler queries to smaller, more efficient models. The second stage features a token-aware verifier that analyzes the sampling probability distribution of each token generated by the small model. This allows it to estimate the model's confidence and decide whether to finalize the response or escalate it to a high-precision cloud-based model. Furthermore, Pro-Router includes an adaptive edge-cloud serving pipeline that dynamically adjusts dispatch sizes to ensure both edge and cloud resources are fully utilized without manual tuning, even with varying network latencies. Extensive experiments across various multimodal benchmarks demonstrate Pro-Router's effectiveness, showing superior routing accuracy and a more than tenfold increase in routing speed. The adaptive serving pipeline also achieves over 75% higher end-to-end throughput compared to existing model routing pipelines, making it a significant advancement for real-time MLLM deployment and cost-efficiency.