TL;DR: Async inference through an AI gateway lets CI build workers submit a long LLM job, get an id back, and poll later, so a 30-second model call stops holding a worker hostage. Here's how I wired it with Bifrost.
Our build workers at Buildkite were each blocked for up to 35 seconds waiting on a single LLM call that summarised failed test output. With a few hundred concurrent builds running through our compute cluster, that's a pile of expensive compute sitting idle on one synchronous request to a model provider. We moved those jobs behind Bifrost, the open-source AI gateway by Maxim AI, and switched them to async submit-and-poll so the worker could get back to running the actual build while the summary cooked in the background.
What async inference actually does
Async inference is a request pattern where the client submits a job, gets an identifier back straight away, and polls for the result later instead of holding the connection open. With Bifrost you set x-bf-async: true on the request and get an x-bf-async-id in return, then poll that id once the model has finished. The docs overview covers the submit and poll lifecycle.
The win is mechanical, not magic. A worker that no longer blocks on a slow upstream can pick up the next build step. On a fleet where each agent costs real money per minute, freeing 35 seconds per build adds up fast across a few hundred concurrent runs.
