Why a machine must be allowed to be wrong
I built meta-science for the All Things Agentic Hackathon, and I wrote this piece for the purposes of entering that hackathon. Code, demo and video are linked at the end — all of it GPL-3.0.
The claim that cannot fail
"Self-improving AI" is the most repeated and least examined claim in our field. Its problem is not that it is false; its problem is that, as usually stated, it cannot be false.
A system that reports its own improvement is indistinguishable from a system that logs the word "improved" and changes nothing. Demos show successes, benchmarks are chosen after the fact, and the reader is asked to extend trust precisely where trust has no purchase. Karl Popper named this failure a century ago: a theory that cannot be
