A real production incident, what the error actually meant, and the fix that stopped it from happening again.
Photo by Chris Ried on Unsplash
Some bugs are polite. They fail the same way every time, on your machine, with a clean stack trace pointing at the exact line. You fix them before lunch.
This was not one of those bugs.
This one only appeared in production, only under concurrent load, and only sometimes. The error code, DB2 SQLCODE -811, is one of those messages that looks like it's telling you something useful while telling you almost nothing. Here's how we tracked it down on a payroll system processing timecards for a large workforce, and what it taught me about writing code that survives concurrency.
