Organizations are increasingly using multiple models to build AI agents in order to find the best balance of performance and cost for each agentic task and LLM call. As we discovered in the 2026 State of AI Engineering report, there isn’t currently a clear winner in terms of adoption among competing models and many organizations are keeping older models in flight despite frequent new releases.
The teams pulling ahead today treat inference like a pipeline, routinely evaluating, benchmarking, and swapping in the right model for each stage—for example, using lightweight models for extraction and tagging and frontier models for synthesis. But without reliable, scalable infrastructure for routing LLM calls across providers and evaluating model performance, it can be difficult to iterate agents quickly, enforce safety and compliance standards consistently, and fail over gracefully when model providers throttle requests or degrade in performance and efficiency.
AI gateways provide an interface for accessing multiple models at a single API endpoint. You can configure gateways to access third-party models from providers like Anthropic and OpenAI as well as your own custom models. By implementing an AI gateway, you can improve security and reliability in your agent environments while making it easier for teams to continuously evaluate and select the best model for each LLM task in their agent loops.
