Friday, March 09, 2007

Thursday morning my phone rang while I was pulling out of the driveway.
One of my co-workers was calling to inform me that I hadn't gotten an email on my BlackBerry since 2am because there was some unpleasantness in the data center. I let him know that I was on my way already, while he listed off systems that would be unavailable until the current crisis had been resolved.
In part, our data center relies on the chilled water system in the office building for our cooling. Since Wednesday night was so meterologically nice, building management had switched over to the smaller chiller. That chiller doesn't provide enough water pressure to properly support the cooling for our sealed, glass-walled server area.
To supplement, we have our own AC unit dedicated for that room.
Unfortunately, Wednesday night the breaker for that unit flipped.
In a few short hours, the temperature in the server room rose to over 120 degrees.
Of course, servers started shutting themselves down, including our email (and BlackBerry) server at 2am, which effectively killed even our notification of the temperature spike. However, people noticed when the servers started vanishing from the network, and we were called.
The door was propped open and a fan put in place to exhaust some of the air before the hard drive platters started to be shaped like Pringles.
Any servers that hadn't powered themselves down were turned off to conserve heat, and we watched the thermostat slowly ease below the 99.9 read out that only reflects how hot it can display.
Long-term casualties included the voicemail server. Last I heard, it was still offline. I try to never check my voicemail anyway, as it just establishes an expectation that I would respond to it, but some people seem to actually use that stuff. I have no idea for what.
The rest of yesterday was spent clearing packing materials out of my cubicle in anticipation of the photographers showing up this morning at 10am.
Our CIO is winning an award for CIO-ing, and there is a chance my cubicle might end up in the background of one of the shots.
Aside from my laptop, at the moment my desk only holds my phone, monitor stand, a file organizer and a new yellow legal pad.
Everything else was unceremoniously dumped into an unused file drawer I hadn't noticed until yesterday. I hope none of it was important, but since most of it came with the cubicle there was no way for me to know. That is, there is no way for me to know aside from reading it, I guess, but that is on my to-do list right after checking and responding to my voicemail.


Ted said...

Headline: CIO Wins Major Award during Data Center Meltdown.

You must be so proud!

Garrick said...

"Ignore the billowing smoke and smile for the camera, sir."