Regular readers know I’m a bit of a nut about Thanksgiving. I’ve declared it, “my holiday.” Last week, I discussed how to enjoy Thanksgiving in IT, and my trip to the Macy’s Thanksgiving Day Parade.
Thanksgiving morning had me up early, tending to stuffing the bird. At 7:40AM, the call IT people dread came,
“We’ve lost all power in the data center.”
Such a simple phrase. Let’s parse it:
- We’ve – it’s a team effort
- lost – no longer to be found
- all power – So all equipment went down hard. You know, it never all comes back perfectly.
- in the data center – oh no!
Suddenly the scene from A Christmas Story went through my head. You know the one…where the dogs eat the turkey?
Thanksgiving was about to be stolen.
What followed was a case study in incident management, with multiple bridges, customer communications, etc.
What did not follow was complaining. Over hours of bridge calls with dozens of people, not once was there a complaint. Sure, everyone acknowledged the day, and family gatherings everywhere were impacted. At my house, the kids had to finish stuffing the turkey and doing all the preparation as I sat (Thanksgiving helpless) in my office.
So around the country, dozens gathered around their virtual work family, and set about the task of restoring a data center. Power was restored quickly, and then the arduous task of livening the entire data center took place.
My son came into my home office. When he was in High School he monitored the website for a major online retailer, and as the one without children often covered holidays, so he understood. “How long will this take?” he asked. “A recovery like this will take days.”
Dessert. There is pie in there somewhere
Everyone kept an upbeat mood as the work progressed. Most people “took a break,” inhaled some Thanksgiving dinner, and returned. My (adult) kids brought me some dessert as they settled in to watch some movies.
What is it about IT people letting them drop everything and focus on the task at hand? What is it about IT people always putting the job before anything else?
You know this event impacted family events everywhere. And yet, without complaint, the work progressed with the majority of systems up before start of business Friday morning.
It’s at a time like this the selfless people of IT deserve a hearty congratulations and recognition of a job well done.
For me and my family, we did a “Make Up” Thanksgiving at a restaurant on Saturday. In some ways, the event made Thanksgiving last a longer time…with even more family time.
Make Up Thanksgiving
My family made it easy on me, and I hope other families did the same. My work family did a great job, and I hope they got to enjoy some of the holiday with their families.
How would your team respond to a unplanned incident like this?