I have a small transformer that turns a one-line spec into a working web app. It is 11.9M parameters, 6 layers, d=384, trained on 4,176 generated programs. Given "a support ticket system with marking a ticket closed, a stats page, searching tickets and creating and viewing tickets" it writes a 1,158-token Python file that compiles, serves HTTP, implements search, stats and toggle, and does not implement comments, edit, delete or category. On held-out feature combinations it does that 5 times out of 6.
Ask it for a book catalogue and it writes an appointment booker.
I built this arm to justify a specific claim, and the claim turned out to be wrong. This post is that retraction, plus the three experiments it took to find out why the model fails, one of which corrected a diagnosis I had already written down and believed.
(I have written about this project once before, on how its verification sweep reports on itself. That post is about the symbolic half of the same system; this one is about the learned half and does not depend on it. The code is in a private research repo, so there is no link. Every figure below comes from a results file or a command I ran, and I say which.)
The claim I was defending
