I was asked to look at a fintech backend where something small but persistent kept happening. Every month when the finance team tried to close the books, the internal ledger and the payment processor's own records were off by a small amount. Not a dramatic amount, sometimes a few transactions, sometimes a few dollars here and there, but enough that nobody could confidently say the numbers were correct. And in a fintech, correct is not optional.
This is called reconciliation drift, and it is one of the quieter problems a payments backend can have. It does not throw an error. It does not crash anything. It just slowly stops being trustworthy, one small mismatch at a time, until someone finally notices during a monthly close and has to spend hours figuring out where things went wrong.
What was actually happening
Once I started digging, the cause was not one single bug. It was a pattern that shows up a lot in systems that grow organically over time, multiple different places in the code were allowed to change a transaction's status or a balance, and none of them were talking to each other.
A webhook handler updated a transaction when the payment provider confirmed it. A retry job also updated the same transaction if the original webhook was ever missed. And an admin tool, built later for handling support tickets, allowed a staff member to manually mark a transaction as settled if a customer complained. Three different code paths, three different points where the same number could be changed, and no single place that recorded why a change happened or which of these three paths actually caused it.
