You ran an eval. The dashboard says 80% accuracy. Now what?
For most teams, the answer is surprisingly manual. Someone exports failures, copies a few examples into a document, writes some notes, maybe creates a ticket or two, and then moves on. By the next eval run, those notes are already stale. The failures have changed, new ones have appeared, and nobody remembers whether a particular issue is actually new or something that has been showing up for weeks.
The bottleneck is not running the eval. It is closing the feedback loop.
Without a structured path from failure → diagnosis → prompt improvement, evals become scoreboards rather than engineering tools.
Recently I ran into exactly this problem while working on a local MLX-based classifier that maps developer work sessions to Jira tickets.
