Monday, July 17, 2006

But anyway, we have this customer. And one day, our sales people sold a report function to them.
This report is generated monthly. By me.
Ok. So I click seven buttons and data is pulled out of a database and placed into the report. The report reflects total system uptime for this customer along with memory and CPU utilization and trouble tickets submitted and concurrent connections. When making it a tidy PDF, it gets called "Sarbanes-Oxley Report" and is delivered to their auditors. I believe we get less than $100 a month for this.
Now, lets lay aside my own bitterness that of all the people on our staff I am the only one qualified to perform the actual audit itself which could make the company a few hundred dollars an hour, easily and deal with the report itself.
Because it broke. A lot.
The problem was that there was no data after June first. I ran the report the exact same way I ran it in May and it just kept blanking on June.
Seriously, there aren't enough steps for me to really mess it up.
I did some digging a discovered that a consultant we had in for a few days loaded a patch on OpenView Reporter at the end of May. Further research showed that there was a well-documented incident where a previous HP OpenView Reporter patch prevented reporting on any date and time not divisible by seven, so I thought this was a likely source of the trouble.
When I brought the possibility up with HP, they called to my attention the fact that our support agreement ended in February and directed my calls and emails to the accounting office.
I explained this to my manager along with the fact that when this happened with Microsoft I put it on a credit card and that it was someone else's turn now.
The strange thing was that OpenView Reporter never threw an error during the report.
Friday I was informed that someone in sales had promised the customer that they would have the report, complete with up to the second data, by 5pm.
I explained that the report wasn't working, and that our usual consultant was traveling all day, and that my latest email about our lack of support agreement had still gone unanswered.
"I don't care. We just need the report."
For some dumb reason, I went back to work on it.
I downloaded a program that let me look at the database in detail. Not being a database person, I don't trust queries. Following the "System Uptime" one, I skipped the Reporter database entirely and went straight to the core production database.
I searched for (customer name deleted to comply with non-disclosure agreement), the dates, "system up", and "system down" messages.
"Zero lines returned."
Odd. I broadened my search. I searched for (customer name deleted to comply with non-disclosure agreement) and the dates -- everything in June.
"Zero lines returned."
At this point I figured I was doing something dumb. I searched for (customer name deleted to comply with non-disclosure agreement) by itself.
"Sixty-two lines returned."
Finding data I'm looking for is good, but it proved that while I was searching correctly and in the right spot, the important data was gone. All that was left in the database was the basic configuration crap. The messages sent that morning were missing, too. And the day before. And last week.
In short, it should be several thousand lines returned. Not (as I got when I ran it a second time) "Sixty-two lines returned."
We can't forget that (due to a drug-fueled deletion incident on Monday) this whole database was restored from tape very recently.
Alright. I'm not ashamed to say I freaked a bit. This publicly-traded company needed this data -- had been promised this data -- for an audit that was to start the following morning. And it wasn't there.
At this point I did the right thing. I went to management and said,"First, the data is missing for this customer. I have no reason to believe that the data exists for any customer. Second, we don't have the data for the report we promised."
"I don't care how it gets done, strip out the data manually if you have to."
"The data isn't there."
He persisted, "I don't care how it gets done, get it done by five."
I wavered,"But I just showed you that the fields are blank. We don't have a support agreement with HP and no one on staff knows this product."
"This isn't getting the report done. I don't care how you do it."
I trudged back to my desk. I stared at my screen.
I looked at the clock.
I combed the database. I cut. I pasted.
I formatted. I spell-checked. I emailed.
I printed.
I walked.
"Here is a hard copy of the report I just emailed. The data is accurate, but not for (company name deleted to comply with non-disclosure agreement). I stole system information from (company name deleted to comply with non-disclosure agreement) and changed the device names. It'll pass an audit."
I left.


Anonymous said...

OMG, I love it :) Did the head fred take the report?


Garrick said...

I think it was delivered. On time is more important than it being fiction I guess.