Building Sluice: QoS-Aware Capacity Governance for Self-Hosted LLM Inference

1. The Problem: When Capacity Becomes the Bottleneck

A self-hosted vLLM deployment runs on a GPU pool of fixed size. That's the fact that changes everything about how you have to think about load.

Once that pool's KV-cache capacity comes under pressure, requests don't naturally understand business tiers. They don't know that one tenant is an enterprise customer with a contract, another is an internal team, and a third is a batch job that can wait. Without something external saying otherwise, the system's behavior under contention is driven by arrival order and backend scheduling — and every tenant, regardless of what they were promised, experiences the same degraded latency, queueing, or failure.

There's no built-in mechanism that says:

Preserve Guaranteed traffic, reduce the work done by Standard traffic, and shed Best-Effort traffic first.

Building Sluice: QoS-Aware Capacity Governance for Self-Hosted LLM Inference

Related reading

Admission Control for Self-Hosted LLMs: Rejecting Requests Before the KV Cache…

LLM Serving Fairness

Inside vLLM: How the World's Fastest LLM Inference Engine Works

Building an Autonomous Budget Gate: Optimizing LLM Costs with Speculative…

Your First LLM API on Kubernetes: From Model to Curl Request

LLM Latency Budget: Make AI Features Feel Fast Without Burning Money