The first agentic feature I shipped looked great in demos. The LLM picked a tool, called it, looked at the result, decided what to do next. Three tool calls, clean output, happy stakeholders.
Then we put it in front of real users.
Within a week we had three incidents. A user retried a request because it felt slow, and we created two support tickets instead of one. A deploy restarted the pod mid-workflow, and the agent simply forgot what it was doing — the user got a half-finished response and no error. Someone needed to approve a refund the agent recommended, but our "approval" was a Thread.sleep waiting on a Redis flag, and the pod that was sleeping got rescheduled. The approval just vanished.
Looking back, every one of those bugs came from the same root cause: we treated tool-calling as a chat completion loop with in-memory state. It works in a notebook. It does not work in production.
This post walks through what changed when we rebuilt the agent runtime as a proper workflow engine on Spring Boot and PostgreSQL — runs persisted in the database, retries with backoff, idempotency keys, human-in-the-loop checkpoints as a real wait state. The full runnable code, Docker Compose setup, and execution evidence are at exesolution.com. This post covers the design choices that matter and how to run it locally.
