A familiar production failure looks like this: an AI support agent reads a ticket, decides the customer deserves compensation, calls the refund tool, and refunds the full annual subscription instead of the $12 add-on. The model did not crash. The API did not throw an exception. The tool worked exactly as designed.
The postmortem usually starts with the wrong question: “How do we stop the model from making bad decisions?”
The better question is: which layer should have stopped the mistake before it became damage?
AI agents fail in many different ways. They misunderstand intent. They create dangerous plans. They pass malformed arguments. They exceed permissions. They loop. They leak data. They take irreversible actions. Each failure mode belongs to a different layer, and each layer has a different job.
If your only defense is a prompt that says, “Be careful,” you do not have a safety architecture. You have a hope.
