Someone Should Be Taking Notes

On Sunday night I stopped billing for my time after submitting over sixty hours for last week.
It got so ridiculous that I actually stopped billing. If there were any condition which would make me seek out medical attention, the willing act of working for free might be it.
It didn't matter too much, though, since by 10pm I was back on the clock trying to fix everything before the end of the outage window or at least before the normals got to work on Monday morning.
While I completely did not care about the root cause of the trauma, I'll detail it here as an object lesson in I.T. Best Practices, Crisis Management, and Dealing With Feelings of Hostility.
My company has a security device to manage remote access. Actually, there are two devices. One is old and crappy, the other is less old and less crappy.
Sadly, both have user accounts on them in giant databases since the migration in December was aborted but the team which configures remote access wasn't notified.
So, in order to ditch the old device, the new accounts on the new device needed to be merged into that older database, then re-imported into the new device.
The new database was backed up and deleted.
This started a chain of unpleasantness which left me hanging out at work for hours and hours all weekend unable to help because my group was completely uninvolved.
See, the deletion process failed most of the way through. Upon investigation, it was determined that the account doing the deletion was deleted during the deletion process . . . whew . . . and then it halted mid-delete since a missing administrator account has no right to delete anything.
That's pretty awful, since you need an administrator account to put the backed-up database back on the device.
The people making this change decided then to find an account with administrative rights which alphabetically followed the administrative account which did the deleting, because that account might still be on the device. For reference, my account name starts with an "x".
Miraculously, they found an administrator. Actually, they found his account. He was out of cellular range -- At the beach.
When they finally contacted him, he gave someone his username and password and work began again.
Until it was discovered that the database they had backed up was corrupted.
Database restoration specialists were contacted and begged to come in and fix what was busted and they set to work pulling out data and rebuilding everything.
After that, re-importing the licensing file to the old machine failed as that file had been corrupted as well.
There was a lot of yelling and cursing.
While the group responsible began the process of manually entering five thousand user accounts, I giggled until I almost wet myself.
Then I joined my team at Waffle House, because that is the only proper response, especially outside of normal business hours when nothing else is open.
Also, waffles rule.