If you serve LLMs on Kubernetes without inference-aware routing, your load balancer is likely wasting inference capacity. Generic HTTP traffic management blindly routes requests, assuming the backends in your cluster are interchangeable. But your model-serving backends are stateful and unevenly prepared to handle any given request. As a result, requests are often routed to the backend that’s not the one best suited to respond.
Migrating to Gateway API gives you a more capable foundation for traffic management and opens the door to inference-aware routing. The Kubernetes Gateway API’s Inference Extension routes requests based on backend serving state, which tends to make better use of cluster capacity and reduce request latency.
In this post, we’ll look at how the Inference Extension works, the routing strategies it enables, and the signals you can use to monitor whether inference-aware routing is behaving as intended in production.
How the Inference Extension works
The Inference Extension improves on conventional HTTP routing for generative LLM workloads by evaluating each backend’s current serving state before selecting an endpoint to receive the request. Standard load balancers were designed for high volumes of uniform web traffic and default to distributing requests evenly. But because LLM inference workloads are highly variable in their request rate, compute cost, and duration, distributing requests efficiently requires assessing the state of each available backend. The Inference Extension looks at signals such as Key-Value (KV) cache state, Low-Rank Adaptation (LoRA) adapter availability, and queue length to identify an optimal target for each request. For example, a backend with a short queue can process a request sooner, and one with a ready KV cache can avoid recomputing the shared portion of a prompt.
