Sleep-time compute: split the budget so a background worker does the predictable thinking while idle — and the user waits far less when they finally ask.
TL;DR: Most agents only think when a request arrives — the user waits and you pay full latency every time. But between sessions there's idle capacity, and many queries are predictable variants of past ones over context that barely changes. Sleep-time compute (Letta; Lin et al., 2025) splits the budget: a background worker pre-answers likely queries while idle, so the foreground serves warm answers instantly and only falls back to a live call on a miss. A freshness check makes sure a changed document never yields a stale answer. In a tiny demo, foreground latency dropped 57% — with novel queries still handled live and stale pre-answers correctly rejected.
Mental model: a prep cook who chops the vegetables before the dinner rush. When orders come in, plating is fast because the prep is already done — and anything that spoiled gets thrown out and re-prepped fresh, never served stale.
The problem: you pay full price at the worst possible moment
Interactive inference spends compute at test time — the exact moment the user is waiting. For one-shot questions over fresh context that's unavoidable. But a huge share of real traffic isn't one-shot: users query the same codebase, the same document set, the same dashboard repeatedly, and the underlying corpus doesn't change between most of those queries. You're re-deriving the same expensive answers on the critical path, over and over.
