NVIDIA MPS Cuts ASR Inference Costs by 75% on EC2

Deploying automatic speech recognition (ASR) models at scale often incurs substantial costs, particularly because individual requests may not fully utilize a GPU's capacity. A new approach leverages NVIDIA CUDA Multi-Process Service (MPS) in conjunction with NVIDIA Triton Inference Server on Amazon EC2 GPU instances to significantly mitigate these expenses. This method demonstrates a potential reduction of up to 75% in GPU infrastructure costs. It achieves this by enabling multiple inference processes to share a single GPU more effectively, thereby maximizing hardware utilization. Crucially, this cost efficiency does not compromise performance. The system is shown to maintain sub-second latency while processing a high volume of requests, specifically 92.1 requests per second per GPU, making it a powerful solution for high-throughput ASR applications.