When an ASR pipeline is pushed to production, the interesting question is not only how fast it runs, but how much throughput you can extract from each GPU before latency starts to break. In the setup described here, that tradeoff was the main lever for reducing inference cost by 75% using NVIDIA MPS on Amazon EC2.
This post is a collaboration between AWS, NVIDIA, and Heidi. It also includes input from Jerron Chua, a Deep Learning Architect at the Generative AI Innovation Center at Amazon Web Services (AWS).
Why GPU sharing matters for inference
If you are running speech recognition at scale, a single model instance per GPU is often not the most efficient use of hardware. There are a few common ways to share GPU capacity:
Time-slicing, where work from multiple processes is interleaved on the same device
