HeraSys Optimizes Collaborative LLM Workflow Serving

The increasing use of Large Language Models (LLMs) has shifted serving systems from handling individual requests to orchestrating complex, high-concurrency, multi-tenant agentic workflows. Current solutions typically focus on optimizing within a single workflow, often overlooking the substantial gains possible from optimizing across multiple concurrent workflows. HeraSys is a new LLM serving system specifically engineered to enhance the end-to-end performance of these concurrent workflows. It achieves this through fine-grained orchestration that identifies and eliminates computational redundancy by merging and reusing structural nodes across different workflows. Furthermore, HeraSys incorporates a load-aware joint scheduling policy that dynamically manages execution order based on both inter- and intra-query priorities. By combining a resource skewing mechanism with adaptive batching and pipeline decomposition, HeraSys effectively mitigates tail latency while maintaining low average latency, ultimately boosting serving throughput by up to 1.85x and reducing P99 latency by up to 2.17x in experiments.