What stress-testing a calculator for AI agents taught me about independent reference data, mutation gates, and sources of truth whose authority can be revoked by evidence.
A multiplication-to-addition bug in regression math passed 389 of 390 tests; NIST's certified reference values detected it. Test coverage alone doesn't guarantee tool correctness—AI systems delegating calculations to dependencies need independent audits.
I gave an AI agent a calculator because I wanted one hard, inspectable point
inside a probabilistic workflow.
The model could interpret the request and explain the result. The calculator
would perform the computation. It seemed like a clean division of labor.
Then I changed one multiplication sign into addition.
Originally published on AIdeazz — cross-posted here with canonical link. The agent passed every unit...
A numeric-citation audit passed 70 of 70 citations across four model families, and a pre-registered equivalence test cleared its…
How I integrated Specmatic executable contracts into an NL-to-SQL engine as a guardrail against AI-agent drift -- across 12…
I let a coding agent write tests as it built. They passed. Then every model change turned a pile of them red. Here is what went…
You Think Testing AI Means Testing the Process? Wrong A Testing Problem My...
6 sessions, 150 standardized tasks, 2 rule formats. The mechanical gate won. Everything else was...
