The Challenge: AI Adoption Across Spotify Spotify's goal for AI is bold, but simple: make it as intuitive and indispensable to daily work as email or Google Docs. And not just for engineers, but for every Spotifier across every function.That vision is driving the AI-transformation strategy, company-wide. The initiative is investing in AI education for 7,000+ Spotifiers, rolling out agentic IDEs to 100% of developers, building shared AI-powered tools for business operations, and establishing the secure foundations for an AI-native workplace. This goes beyond a tooling rollout or a training campaign: it's a fundamental shift in how the company operates.But ambition at this scale creates an infrastructure problem. Gen AI-powered features are increasingly integrated across various parts of Spotify, including external areas like music, podcasts, ads, and search, as well as internal functions such as legal, social, marketing, and finance tools. Behind each of those features, there's a team that needs to talk to an LLM. A year ago, that meant each team was on its own. One squad would integrate directly with OpenAI's API, another with Vertex AI, another with our first-party Hendrix models. Every team resolved the same problems: how to authenticate, how to handle rate limiting, how to track costs, how to fail over when a provider goes down. There was no centralized observability, no consistent error handling, no uniform safety policy, and large governance overhead as well.It worked when AI usage was a handful of experiments. It stopped working as hundreds of consumers began onboarding with fragmented LLM inference requirements — from AI-powered search to ad generation to DJ narration to internal developer tools. Every new integration meant weeks of boilerplate. Every outage at a model provider triggered a scramble across unrelated teams. And when leadership asked how much we were spending on LLMs, nobody had a single answer.Without shared infrastructure, Spotify risked exactly what the strategy was created to prevent: fragmented adoption, duplicative tool development, and operational inefficiencies. We needed a gateway.Why Kong: Building on Proven API Infrastructure One of the five strategic pillars for the AI-transformation strategy is "Building the AI Engine" — creating the shared, secure foundations that allow teams across the company to move fast and safely with AI. A major component of that engine is the AI Gateway, which launched in May 2025. When we evaluated how to build it, we had a clear build-vs-buy decision to make. We could write a custom proxy from scratch, or we could stand on the shoulders of something battle-tested.Performance benchmarks confirmed the choice. Kong demonstrated ~40ms routing overhead and was expected to scale beyond 1,000 RPS per instance — more than enough for our traffic patterns. And because Kong is fully self-hosted, all data stays within Spotify's infrastructure — a non-negotiable requirement for a system handling every LLM prompt in the company.We started with Kong OSS with a clear path to enterprise features if needed. As we scaled, we realized that there were features we required that weren't available open source. Because we were running business-critical traffic through the AI Gateway, we needed enterprise-grade security and support. That was when we talked with the Kong team and made the move to the enterprise version. Here's what made it the right fit:Hybrid mode fits our deployment model. We run on GKE, and Kong's separation of control plane (configuration management, backed by PostgreSQL) and data plane (stateless proxies handling traffic) mapped directly to how we wanted to deploy. The control plane manages config and connects to CloudSQL. The data planes scale independently and handle all client traffic. No shared state between proxy pods means we can scale horizontally without coordination.Declarative configuration fits our GitOps workflow. Kong's decK tool lets us define every consumer, route, service, and plugin in version-controlled YAML. Every change goes through a pull request, gets validated automatically, and rolls out through our CI/CD pipeline. No one clicks buttons in a UI to change production routing.The plugin ecosystem gave us extensibility without forking. Kong ships with a rich set of bundled plugins, and we can build custom ones in Lua. We needed authentication, guardrails, and provider-specific logic — the plugin model let us compose these cleanly instead of building a monolith. We have currently developed plugins that provide security and safety capabilities.AI-native capabilities accelerated our timeline. Kong's ai-proxy plugin handles multi-provider routing and request transformation. The Prometheus plugin's ai_metrics: true flag gives us token-level telemetry out of the box. We didn't have to build AI-specific observability from scratch.A standard interface for model switching. One of the deciding factors was Kong's ability to provide a single API format while transparently routing to different providers — first-party Hendrix models, OpenAI, or Vertex AI. Teams integrate once and can switch providers without code changes. This was critical for the early use cases like AiKA (Spotify's internal AI knowledge assistant) and our Prism models that we needed to onboard first.Architecture: A Gateway Purpose-Built for AI TrafficAI Traffic is Not Web TrafficBefore diving into the architecture, it's worth explaining why a general-purpose API gateway isn't enough for AI workloads. AI traffic has fundamentally different characteristics: