I'm building LiveSuggest, a real-time meeting assistant. It listens to your call and shows you written suggestions while you're still talking: a clarifying question to ask, a point you forgot to make, the definition of a term someone just dropped. No bot joins the call. Everything runs from your own browser.
The whole product lives or dies on one number: how long between someone saying something and a useful suggestion appearing on screen. A suggestion that arrives after the conversation has moved on isn't just useless, it's a distraction.
So I assumed the language model would be my enemy. LLMs are the slow, expensive part, right? I instrumented the entire pipeline in production, expecting to spend my life fighting token latency.
I was wrong about where the time goes. Here's what a week of production metrics actually showed me.
The pipeline
