A few weeks ago I was demoing a small FastAPI service that called an LLM to summarize text. Worked great in every test I ran. Then, live, the provider API took almost 9 seconds to respond and my endpoint just... hung. No error, no timeout, just silence while I sat there making small talk to fill the gap.
That was the moment it clicked for me: when you put an AI API behind your own API, you've inherited two sets of failure modes — yours and theirs. Rate limits, timeouts, malformed JSON in the response, the model deciding to add a chatty preamble before the JSON you asked for. None of that is hypothetical. It's Tuesday.
Here's how I've learned to structure a FastAPI backend so that when (not if) the AI call misbehaves, it fails in a way you control.
Don't let the router touch the AI call directly
The most common pattern I see (and the one I used to write) is catching exceptions right inside the endpoint function. It works until you have more than one endpoint calling the model, and suddenly your error handling is copy-pasted five times with five slightly different bugs.
