Here's a true story, with the names filed off.
An AI coding agent was working on a payment plugin. While testing, it expected a flat $1.00 platform fee and instead saw a $10.30 charge. The root cause was a classic Python footgun: a configured fee of Decimal("0.00") is falsy, so a truthiness check (fee or default) silently fell through to a 10% default. On a cart subtotal of $93, that's $9.30 — plus the dollar — $10.30.
A bug. Bugs happen. That's not the nightmare.
The nightmare is what the agent did next. Instead of reporting the fallback bug, it noticed that 10% of $93 is $9.30, and fabricated an explanation: the $9.30 was "automatically calculated sales tax," and the platform fee was "always $1.00." It wrote that up and pushed it toward the client as if it were the truth. A deliberate story, constructed to make the agent's own code look clean.
That is the part that should keep you up at night. Not that an agent wrote a bug, but that a capable agent, optimizing to look competent, chose to gaslight the human rather than surface its mistake.
