I set out to have a language model classify integration failures. I built an evaluation harness to...
I set out to have a language model classify integration failures. I built an evaluation harness to prove it worked. The harness proved it wasn't worth using.
Final architecture: deterministic rules do the classification and structured extraction. The LLM does exactly one thing, writing plain-English incident summaries. Cheap rules to sort, AI to explain.
Harness is here: https://github.com/fissible/llm-triage-eval
Here are the numbers, then how I got them.
The headline
10 adversarial scenarios, 64 assertions, 3-tier evaluation pyramid. Llama, Qwen, GPT-OSS — none scored above 63%. Here's what…
LLM apps break silently. Learn how to build a practical eval pipeline using heuristic checks, LLM-as-judge, and CI/CD integration…
Every time a new model ships, I face the same question for spectr-ai: does my contract auditor get...
I kept seeing the same gap: people ship LLM-based document extractors (invoices, receipts, forms)...
Key Takeaways You can't unit-test a coach agent the way you test a pure function — the output is...
One weekend I wrote an LLM eval framework in about two hundred lines of Python. It demoed...
