Friday, May 18, 2007

Holy crap!

A lot of times I talk about things being "on fire" or "going up in flames" at work, but even in the worst cases this was rarely literal.
Yesterday was different.
The lights flickered just after noon. They do that, but then the power totally failed.
There was a collective groan and the server room door was thrown open immediately.
We have back up power for most of an hour for the servers, but the cooling goes away completely instantly and the temperature starts to spike within minutes.
There was a sound like gunfire from outside, which is odd because we are on the 28th floor.
Across the freeway and about a block down, we saw a thick column of black smoke.

Photo Sharing and Video Hosting at Photobucket

A power substation, the one servicing seventeen square blocks, including ours, was actively exploding.
And it was loud where we were, a few blocks away and 28 floors up.
The Mexican restaurant next to it looked like a freshly kicked ant bed as the lunch crowd swarmed the parking lot and explosion after explosion and flames two stories high told us power wouldn't be coming up any time soon.
Seriously, the smoke was thick and the electricity was arcing all over the taller . . . metal . . . thingies.
It was awesome.
And then we had to turn off all the servers before they cooked themselves.
They gave us two hours to evacuate the building and we finished the shutdown with seven minutes to spare.
Every traffic light in seventeen blocks was out.
The parking garage was backed up to the sixth floor.
I park on the sixth floor.
We waited in the lobby, my friends who park on six and seven with me.
And finally, I went to my car.
Brian (the network guy from work and an awesome guildmate in WoW) ran up, waving at me as I started to back out of my spot.
He gestured up. I looked.
The parking garage lights were on.
I killed the engine and we headed back upstairs to turn everything back on after we confirmed the complete restoration of electrical service.
It took a few hours.
That fire was big.