A model gateway is a service every model call in an organisation goes through, so that authentication, routing, quota, failover, logging and key custody happen once instead of in forty codebases. It is a straightforward piece of engineering with two genuinely hard parts — correct streaming and distributed quota — and a maintenance burden that is larger than the build. This page describes the architecture, and then states the costs.
What a gateway is for
The case for one becomes obvious at a particular size. Once several teams call models independently, five things are true at once: nobody can say what the organisation spends, provider keys are distributed across services and CI systems, switching providers means editing every repository, there is no consistent record of what was sent to a third party, and every team has written its own retry logic with different bugs.
A gateway addresses all five by being the single egress point. That is its real definition: not a proxy, but the place where organisational policy about model use is expressed once and enforced.
It also introduces a hard dependency — a component that, when down, stops every AI feature in the company. That is the central trade, and it means a gateway must be engineered to a higher availability standard than most of the services behind it. Design for it to fail open where policy permits: a client library that can call a provider directly with a break-glass credential when the gateway is unreachable turns a total outage into a degraded mode, at the cost of a credential that must be audited.
