How llm-d Prefix-Cache Routing Made Qwen 7B on EKS 2.3x Faster

Introduction I wanted to benchmark how much the routing layer matters for LLM inference...

Raccontata dadev.to