When to Declare an Incident

I was sitting with a client recently having a project discussion when a problem with email was called in.

No big deal.  Large companies have problems every day.

A few minutes later another interruption, this time for phones in a remote office.  OK, this is why we have staff.

And then another issue, this time with remote access.

The client was calmly processing these facts, and continuing the conversation.  Perhaps they felt obligated with me being there.  I interrupted, and said, “All these problems.  Something larger must be going on.  Should you declare an incident?”

Large IT shops are very familiar with large scale incidents, and have well-honed incident management processes.  When there are major issues (like a total processing failure, or a major business impacting outage) the incident processes are automatically implemented.

It’s incidents in the grey where sometimes companies are hesitant invoke the incident management process, often involving many people with full notifications to the business.

While there are books written around how to do incident management, when to declare an incident isn’t uniformly understood.  Why?  Because in IT every day we have a multiple of issues dealt with in the normal course of business.  Declaring an incident is often viewed as a “big deal.”

It is a big deal.  The best and brightest drop what they are doing and focus on the issue. 

The catastrophic failures are easy…instant incident.

On others, often tickets are being opened on help desk(s) and routed for resolution.  If you’ve ever been around a help desk, you know if one minute they are busy and the next they are slammed….there is something going on.

When considering declaring an incident, POTENTIAL BUSINESS IMPACT is my metric.

So, one desktop down is important, 100 desktops down in a call center is a big deal.

A one person sales office having a phone issue is bad, and a 250 person HQ being without phones is really bad.

The way to minimize business impacts is through contingency planning.  If one facility is down, the business/traffic is routed somewhere else. 

Large companies do this as a matter of course; smaller companies often don’t feel they have the mass to successfully pull it off.

Declaring an incident should be celebrated as a way to get others to help quickly mitigate business impacts.

When have you “declared an incident?”