Pr3++yG33kyTh1ng

Over the weekend a firewall hole used for testing was accidentally left in place. As a result, some enterprising spammer spent the whole weekend relaying junk email through our mail server. Thousands and thousands and thousands of emails.
Of course, it all got lodged in our outbound filtering system and stopped all mail by Monday morning.
On the bright side, I know where to get cheap prescription drugs.
Since I personally have nothing to do with the firewall and very little to do with our email system, I came out of the meeting as the only one un-fussed at. Looking good by comparison is the same as looking good based on individual merit. Not that any fussing was of a level I'd deem noticeable anyway.

Basically, my boss turned on his speakerphone and addressed the offending team member (who was not only not responsible for the firewall but had been working on the mail issue from home since 4am) by telling him how shocked and disappointed he was in the whole situation. He did not curse. He did not raise his voice.
And technically, even though my cube is about 10 feet from his open office door, the conversation was private.

By 10:30, things were going back to normal in the email world, so my boss called a meeting.
He apologized (three times over about 10 minutes) to the admin he had called out over the speakerphone and to the rest of us who may or may not have heard it.
I chose to not tell him that I've been personally insulted and demeaned at the end of a successful project and that contact from management during an actual crisis has often stopped just short of physical shaking. If he thinks that is the line I'm not going to be the one who argues to re-draw it someplace angrier.

And then I went to work on our old Citrix farm. I've avoided it like it has an open dripping wound so far, but there are applications still in use on the old girl and they are (understandably) broken.
Our new farm is load balanced and up-to-date and accessible from (literally) anywhere on the planet. The hardware is state of the art and the event logs are remarkably free of angry red marks of Microsoft pain.
The old farm is two wheezing servers running an outdated version of Citrix. Also, due to an error in the licensing protocol, you can't actually connect to them through Terminal Services. This means that to work on one, I need to plug in a mouse and keyboard and monitor. Only then, they don't show up. Even using USB hardware the operating system won't find them without a reboot.
So I power cycled the first server blind, with no way to know if users were on it, and the keyboard still didn't work. I needed to launch a session on it to initialize the drivers, only there is no real way to launch that session.
In the end, I found that the only way to gain access was to launch a third party application and break out to a desktop. That done, I discovered that the problematic applications were deployed on the other server. The other server that did not share the hackable application.
Due to lack of documentation, I started a full forensics job from the accessed server to the running production server I have no way to get into.
I kept picturing the swamp monster in the stump in the Flash Gordon movie. Timothy Dalton was pretty enough. I'm at least as pretty as Timothy Dalton.
So I dug around in a dying server full of users most of the day yesterday, pulling applications and *.dll files to the safety of the new farm and telling the Desktop Group to expect the calls as people start to have applications vanish.

Plans for today include finding the ex-employee who left me this mess for a fight to the death on top of a floating, twisting, spiked platform.