Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine | Amazon Web Services

Running large language model inference at scale forces a KV cache trade-off: oversized GPU instances or slow time-to-first-token. This post builds a tiered KV cache on Amazon SageMaker HyperPod that extends the cache into a shared, distributed NVMe pool with Curvine, so replicas reuse cache at near-local-disk speeds on cost-efficient instances.

Raccontata daaws.amazon.com