Last Monday I shipped the shipping-tool prompt from Part 9 to 5% of live traffic. It had won the pairwise gate 24 to 7 with 9 ties, the tool discipline diff was clean, and the money-path cases read fine by hand. I went for lunch confident.
At 6:42pm the canary dashboard disagreed. P95 latency was up 38%. Tool calls per conversation had climbed from 3.1 to 5.4. The reworded tool description I was so proud of had taught the agent to call the shipping tool twice per turn, and in real conversations, which run much longer than my 40 test cases, every extra call doubled the wait. I rolled the split back to zero in nine minutes, and the numbers returned to baseline by 7:15.
The 40-case dataset from Part 8 could not have caught this. Its transcripts are short by design. The pairwise judge from Part 9 could not have caught it either, because it judges two responses, not the whole session cost. Only traffic could, and traffic only talks to you if you route a slice of it first.
This part is the production runbook I promised at the end of Part 9: canary traffic splits, automatic fallback when a model degrades, and cost caps that stop a prompt regression from becoming a bill regression. The agent is the same e-commerce assistant from Parts 1 through 9: nine tools, conversation memory, the supervisor, and the human-in-the-loop checkout gate. I have been building production AI agents with Spring Boot and Spring AI for over a year, and every number below is from the rollout as I actually run it.
