Writing Evals for an LLM Security Tool: How I Know It Didn't Get Worse

Every time a new model ships, I face the same question for spectr-ai: does my contract auditor get better or worse on the new model? "Vibes" is not an answer when the tool tells people whether their code is safe. So I built evals. Here is how I test an LLM that produces fuzzy output, and why a handful of labeled examples beats a gut feeling every time.

The problem with testing LLM output

Unit tests assume deterministic output. LLM output is not deterministic, and even when it is correct it phrases things differently each run. You cannot assert output === "reentrancy on line 12". The model might say "the withdraw function is vulnerable to reentrancy" or "external call before state update in withdraw()."

So I do not test the text. I test the finding. Did the model identify the vulnerability class on the right function? That is a yes/no I can check, and it survives rephrasing.

The eval set

Writing Evals for an LLM Security Tool: How I Know It Didn't Get Worse

Other newsrooms on this story

Related reading

Building Evals That Don't Lie: How to Make AI Evaluation Reliable in Production

I Built an Adversarial Eval Framework and Attacked 5 LLMs — Every Single One…

LLM Evals For Developer Tools: Useful, Correct, Safe

The Auditor's AI Workflow: How I Use LLMs Without Trusting Them

An open source LLM eval tool with two independent quality signals

A RAG evaluator that admits what it can't judge