Every morning, before anyone on the team has looked at a dashboard, a Claude Routine has already read around 800 of the previous night’s conversations from Seer, Sentry’s AI agent for triaging and fixing errors. It flags the ones that look broken, and files tickets for anything new. By the time we sit down with coffee, the triage is mostly done.
The problem
Before this routine existed, checking on our agent’s behavior meant a custom, one-off flow for pulling data out of our hosted Langfuse instance. Querying options were limited, so most of it came down to hand-rolling a parser for the raw JSON in each trace. Someone had to remember to trigger the analysis and do it manually. Automating this was an option, but we never got around to it due to friction. Plus, we wanted to send the data to the other tools we use day to day, and the current setup didn’t have that kind of connectivity.
We were looking for a mix of things. Tool call failures, hallucinated outputs, latency spikes, agent loops that went on longer than they should have, and cost anomalies. All of the above, really.
We had three main goals. Improve reliability by cutting down tool errors, keep an eye on the long tail of unusual runs (the ones that were unusually expensive or made far more tool calls than a typical conversation), and sample the conversations that looked good to actually confirm they were good, not just assume it.
