I got locked out of my own production server. Not "forgot the password" locked out — both SSH and the hosting panel's browser console rejected me, on a box running several live apps, for hours.
Here's what actually happened, and the two lessons that came out of it.
The wrong assumption that cost the first two hours
My hosting provider's browser-based rescue console kept rejecting root's password, even right after resetting it through their panel. I assumed the reset was broken and spent ages fighting it.
It wasn't broken. On a standard Ubuntu cloud image, the account the rescue console logs in as is ubuntu, not root — a convention from how cloud images provision the default user. Root's password can be perfectly correct and still be useless there. On top of that, my ubuntu account turned out to be locked (passwd -S showed L) since the box was created, completely unrelated to any password value.
