FleetSieve Optimizes LLM Fleet Configuration with SLO-Aware Profiling.

Configuring large language model (LLM) serving fleets, particularly in terms of tensor-parallel (TP) degrees and replica counts, is a complex challenge. The optimal configuration is not monotonic with TP and can shift under varying loads, making exhaustive profiling computationally expensive. This paper introduces FleetSieve, a novel profiling approach designed to make this process more efficient and decision-critical. FleetSieve intelligently selects which configurations to measure by modeling capacity and tail latency jointly, focusing on measurements that are most likely to influence the final resource allocation decision, which is coupled with Service Level Objectives (SLOs). It iteratively compares conservative and optimistic allocations, stopping when the decision gap falls below a specified tolerance. In experiments with a 31B-parameter model on H100 GPUs, FleetSieve achieved the oracle aggregate decision using significantly less GPU-time than uniform random profiling (6.9% less in a fixed comparison, 5.4% mean saving across random reveal orders). It also successfully avoided selecting configurations that violated SLOs, demonstrating its ability to make robust, performance-aware decisions. Incorrect sparse-profile decisions, by contrast, could lead to substantial losses in requests per second and fulfillment rates.