Optimizing generative AI inference deployments requires benchmarking endpoints, evaluating instance configurations, and iterating on deployment settings. The Amazon SageMaker Python SDK v3 now exposes generative AI inference recommendations in Amazon SageMaker AI directly in your notebook workflow. These recommendations are also accessible through the Amazon SageMaker AI UI and Boto3 APIs. With this release, you can benchmark an endpoint, generate data-driven deployment recommendations, and deploy the recommended configuration directly from a notebook using the Amazon SageMaker Python SDK v3.

In this post, we demonstrate how to use the new SDK interface for the end-to-end workflow to optimize generative AI inference deployments.

Benefits of generative AI inference recommendations in Amazon SageMaker AI

Generative AI inference recommendations in Amazon SageMaker AI automate inference optimization by:

Benchmarking a live Amazon SageMaker endpoint against a synthetic or real-traffic workload, measuring throughput, time-to-first-token (TTFT), end-to-end latency, and more.