Most platform teams didn't set out to build "AI infrastructure." They already had Kong running in front of hundreds of REST services on GKE, handling auth, rate limiting, and observability the way an API gateway is supposed to. Then product teams started shipping LLM-backed features, then RAG pipelines, then autonomous agents that call tools on their own and suddenly the gateway layer that worked fine for CRUD traffic started showing cracks. A single chat request can cost 50x another one depending on the model and prompt length. A single "user request" might now fan out into a dozen tool calls made by an agent with no human in the loop. None of that maps cleanly onto request-count rate limits or static routing rules
This is the problem Kong's AI Gateway capabilities are built to solve, and GCP is a natural place to run it: GKE for the gateway itself, Cloud Load Balancing at the edge, Vertex AI and Model Garden as first-class model backends alongside OpenAI, Anthropic, and others, and Memorystore for Redis backing distributed rate-limit state. Below is a practical walkthrough of turning an existing Kong deployment into a real AI gateway on GCP covering token-aware traffic management, Model Context Protocol (MCP) support, and the harder problem of governing agentic traffic that doesn't behave like normal API calls.
