TL;DR: Async inference through an AI gateway lets CI build workers submit a long LLM job, get an id back, and poll later, so a 30-second model call stops holding a worker hostage. Here's how I wired it with Bifrost.
Our build workers at Buildkite were each blocked for up to 35 seconds waiting on a single LLM call that summarised failed test output. With a few hundred concurrent builds running through our compute cluster, that's a pile of expensive compute sitting idle on one synchronous request to a model provider. We moved those jobs behind Bifrost, the open-source AI gateway by Maxim AI, and switched them to async submit-and-poll so the worker could get back to running the actual build while the summary cooked in the background.
What async inference actually does
Async inference is a request pattern where the client submits a job, gets an identifier back straight away, and polls for the result later instead of holding the connection open. With Bifrost you set x-bf-async: true on the request and get an x-bf-async-id in return, then poll that id once the model has finished. The docs overview covers the submit and poll lifecycle.
The win is mechanical, not magic. A worker that no longer blocks on a slow upstream can pick up the next build step. On a fleet where each agent costs real money per minute, freeing 35 seconds per build adds up fast across a few hundred concurrent runs.
Why synchronous LLM calls stall a build fleet
A build agent is a finite resource. When it makes a blocking HTTP call to an LLM and the provider takes 30-plus seconds, that agent is doing nothing but waiting on a socket. Multiply that by every failing build wanting a summary, and you've quietly turned your model provider's latency into your queue depth.
We saw exactly this. P95 latency on the summariser sat around 28 seconds, and during a flaky-test storm the build queue backed up because agents were parked on those calls. The compute was healthy; the scheduling was wrong. The fix is to decouple "ask for a summary" from "wait for a summary."
Wiring async submit and poll
The change was small. We added two headers and split one blocking call into a submit step and a later poll step. The Bifrost endpoint stays OpenAI-compatible, so the request body didn't change at all, which is the point of a drop-in replacement.
# Submit: fire the job, return immediately with an id
curl -s -X POST http://bifrost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-bf-async: true" \
-H "x-bf-dim-team: build-platform" \
-d '{
"model": "openai/gpt-4o-mini",
"messages": [{"role": "user", "content": "Summarise this failed test log..."}]
}'
# response carries x-bf-async-id: job-8f21
# Poll later, from a separate build step, once the agent has moved on
curl -s http://bifrost:8080/v1/chat/completions \
-H "x-bf-async-id: job-8f21"
We submit at the start of the post-build hook, run the rest of the cleanup, then poll near the end. By the time we poll, the summary is usually ready, so the agent almost never waits. The x-bf-dim-team header tags the request with our team name, which Bifrost auto-forwards to logs, traces, and Prometheus so we can see which team's jobs are driving spend.
Keeping costs and failures visible
Async jobs are easy to lose track of, so observability matters more, not less. With Bifrost the custom x-bf-dim-* dimension headers flow straight into the observability layer, which writes asynchronously and adds under 0.1ms of overhead per the benchmarking docs. That let us build a Grafana panel keyed on team and job type without instrumenting our own code.
Failover still applies to async jobs. We kept automatic fallbacks configured so that if our primary provider returns 502s, the gateway retries against a secondary before the job id ever comes back failed. On the throughput side, a single instance sustains 5,000 RPS at 100% success with roughly 11µs of gateway overhead on a t3.xlarge, per the published benchmarks, so the gateway itself was never the bottleneck in our queue.
Trade-offs and limitations
Async is not free. You now own a polling loop and the job ids it depends on. If a build agent dies between submit and poll, you need those ids in durable storage or the result is orphaned. We push ids into the build's metadata so a retried step can recover them.
On the Bifrost side, self-hosting carries real operational weight. A production deployment needs Postgres backing it, which is one more stateful service for my team to run and patch. Clustering for high availability is an enterprise feature, not part of the open-source core, so a single-node deploy is a single point of failure you have to plan around. The ecosystem is also younger than LiteLLM, so there's less community Q and A when you hit an edge case. None of that was a dealbreaker for us, but plan the operational side before you commit.
Wrapping up
Switching CI summarisation to async inference through Bifrost took the blocking time off our build agents and stopped a slow model provider from setting our queue depth. The headers are simple, the endpoint stays OpenAI-compatible, and the spend stays visible per team. If you run LLM calls inside a build fleet and your agents are parked waiting on them, async submit-and-poll is worth a look: https://getmaxim.ai/bifrost/book-a-demo