Someone on your team built an LLM feature. Maybe it's an inline code-suggest. Maybe it's a "fix this PR comment" button. Maybe it's a full agent that opens pull requests on its own. The demo worked. The screenshots were good. You shipped it.
Now a real user gives it a real codebase, and you have no idea whether it's getting better or worse week to week.
That gap, between "it worked in the demo" and "we can prove this is improving," is what evals are for. And in 2026 we are still surprisingly bad at it. We point at SWE-bench Verified scores like they're the same number as "does it work on our repo," we trust LLM-as-judge scoring more than the literature says we should, and we mistake low-latency token streaming for usefulness. This piece is a practical map of how to measure the three things that matter for a developer tool: is it useful, is it correct, and is it safe.
What Makes Dev-Tool Evals Different
A lot of generic LLM eval advice was written for chatbots, where the output is a paragraph that a human reads. Developer tools sit on the other end of the spectrum. The output is usually a diff, a file, a search result, a piece of structured JSON, or a side effect: a closed issue, a green build, a published artifact. That changes almost everything about how you evaluate.
