Originally published on Loop & Retry — field notes on building LLM agents that survive production.
Here's the pitch for best-of-N: instead of trying once and retrying on failure, fire off N attempts at the same task simultaneously and keep whichever one finishes first (or scores highest). You've turned a serial wait into a parallel one, so your tail latency drops — no more waiting through however many retries it takes before one succeeds. The catch that doesn't show up in the pitch: you pay for all N attempts every single time, whether you needed them or not, and when that payment stops being worth it depends entirely on a variable most teams never measure — whether your attempts actually fail independently of each other.
The two strategies, priced the same way
The retry-budgets post modeled sequential retry cost: try, and on failure, try again, accumulating the failed attempt's tokens into the transcript each time. Best-of-N is structurally different — there's no accumulation, because the N attempts don't see each other. Each one starts fresh from the same prompt and runs to completion independently. That makes the accounting simpler, which is exactly what makes the tradeoff easy to misjudge.
