Gary L Kelley - Journal

To:     All Personnel

From:     Information Technologies

Subject:     System Outage

As Infrastructure & Operations professionals we all delivered these emails or memos to our users. It pains us when we have to communicate this information, but how and what we communicate ultimately defines the respect we receive from our users and senior management.

I am reminded of an outage that occurred while I was at a seminar in New York City. Our trading system was impacted while data was being replicated to our Disaster Recovery systems. We could not assess the impact of failing over, so we attempted to fix the problem. I was immediately paged and alerted the CIO. It appeared we would not be able to restore the trading system in time for the start of day so I immediately packed and got the 6:30AM flight out of New York. On my way to the airport I informed the team I would speak to the CEO as soon as I landed. I got on the plane (still on the same conference call from 3:00AM), and who do I see 10 rows in front of me, but the CEO. I couldn’t get to him on the flight so I figured I would catch up to him in the airport. Luckily as I was running through the terminal I was informed all systems were operational and business would proceed as usual. I caught the CEO and, relieved from the news I just heard, joked how it was funny we were on the same flight.

Unfortunately, situations do not always end up this way, and you must find ways to communicate bad news to your users. Over the years, the following guidelines served me well:

Determine the sender of the message – The sender of the message will also be the person to receive any questions or comments about the incident and status. A message sent from a person, instead of a generic mailbox, will carry more credibility, but will yield more questions.

State the problem in the first paragraph – Let people know what happened in the first sentence. Use the second sentence to communicate business impact. Never use any technical terminology and never describe systems by their internal IT name. Always describe systems by the business functions hosted on the system.

Identify functional business systems – The first paragraph notifies users as to systems not available. The second paragraph lets them know systems that are available. Users want to know if they can do their job and your communication must not be ambiguous in this context.

Give estimated time for recovery or time of next status message – People want to feel informed and in some control. Frequent status updates helps to achieve this goal. If an estimated time of recovery is unavailable, let people know when the next status notification will be issued. Over time, you’ll be able to either have accurate estimates, or will get a “gut-feel” for the length of outages.

Designate a point of contact – Users will have questions. Give them contact information. The Help Desk would be our first choice. Questions will also be directed at Desktop Support personnel as they have the most frequent contact with users. Do not forget to notify them before you notify your users.

Do not apologize if it was not an internal problem – Sometimes, third-party hardware and software fail. We try and prevent this from happening, but there are times when it is beyond our control. Only apologize when you have something to apologize for.

As problems become more complex and require larger numbers of IT personnel, we recommend using a dedicated Problem Communication Manager for generating both internal IT and external updates. This will simplify the job of the Incident Manager and provide better service to users.

Defining problem communication processes, people, and templates delivers a higher level of service. Users will appreciate the communication and it will be one less task for IT personnel involved with the problem.