Want email updates from me?
Want more unvarnished truth?
Looking for something? Look here!
What I'm saying now
What you're saying...
I think tag clouds are pretty, and not to be taken overly seriously
111 Chop House 75 on Liberty Wharf 9/11 A Broth of a Boy ABCs Abiouness accountability activities alcohol Allora Ristorante Analysis Angry Hams ANSI/TIA 942 Anthony's Pier 4 Apple Application Armsby Abbey Arsenal Arturo's Ristorante Ashland AT&T Audio Automation baby Baby Monitor babysitting Back To School Bad News Bangkok Thai Bar Bay State Common baystateparent BBQ BCP BeeZers Before I die I want to... behavior Big Bang Bike Bill of Rights Bistro Black Box BlackBerry Boston Boston Marathon boundaries Boyston BPO brand Breakfast Bridge Bring Your Own Technology Budget Burlington Burn Burrito buyer BYOD Cabling Cambridge Camp Campaign career Casey's Diner Castle casual cCabling Cell Phone Central Square Change Management Cheers Chef Sun ChengDu Chet's Diner Children Chinese Christmas Christmas Families Holiday CIO Cloud coddle collage College College Acceptance co-lo Co-Location Co-Location Tier Power Cooling Comfort Food Control Country Country Kettle Crisis customer dad Dad Phrases damage daredevil Data Center Data Center Design Davios Day Care Dead Death declaration Del Frisco's Design Desktop Video dinner Disaster Recovery Divorce Do Epic Shit dodgeball Downtown Crossing DR driving Droid Easter Economic Kids Edaville Education Elbow Night Elevator Employee Engagement Erin Etiquette Evaluation events Exchange Expiration Dates Facebook Failing family Fatherhood Favorite things Flash Flemings Fogo de Chão Food Hits and Misses Format Foundry on Elm Foxborough Frameworks fraternity Fraud French Fried Clams friends fun Fusion Generations germs Girl Scouts girls Global Go/No Go GPS Grafton Grandchild Grandpa Harry's hazing Healthcare Healthy Choices while Dining Out Help Desk Hisa Japanese Cuisine Historic holiday Home hope Horizons hose Hot Dog Hurricane IIT Assessment incident Indecision Indian Infrastructure Inn Innovation Internet Inventory Management iPhone IT IT Assessment IT Satisfaction Italian Jack Daniels Jakes Restaurant Janet Japanese Jazz Joey's Bar and Grill JP's Khatta Mitha kickball kids Laid off Lakes Region Lala Java Leadership Learning legacy Legal Harborside Les Zygomates L'Espalier Liberty Wharf lights out Linguine's loss Love Lucky's Cafe M&M Macys Thanksgiving Day Parade mai tai Managed Application Services Managed Services managers Mandarin Manners Mark Fidrych marriage Mary Chung mass save Maxwell-Silverman Mediterranean meetings Memorial Day memory Mendon Mergers Mexican MiFi Migration Ming III miss MIT MIT CIO Symposium Mobility Moes Hot Dog Truck MOM money Mother Moving on Name neanderthal neighborhood Network New York Marathon newborn Northborough Not Your Average Joe's Nuovo Nursing On-Call Operations Operators Oregon Club Organization Pancakes Pandemic Parental Control Parenting Patch Peeves People Perserverance UMASS growth Play Plug and Run Predictable Pride Problem Process Production program Project Management propane PTA. PTO PUE QR Quick Response Rant Real Estate Realtor Recognition Red Rock Resiliency Respect restaurant Restaurant Guy RFP ribs Ritual Root Cause Analysis Sam Adams Sandy Sapporo savings School Sea Dog Brewing Company Sea Dog Steak and Ale Seafood Seaport Security Sel de la Terra Service Service Desk Service Indicator Light sharing SHIRO Shit Pump Shriners SHTF Simplification Skunk Works Skype Sleep sleepovers Sloan Smith & Wollensky soccer Son SOP sorority spanking Squarespace staffing Starbucks Status Reporting Steak Steve Jobs Storage Strategy stress Summer Sushi swimming Tacos Acalpulco teacher Technology Teen Telephony Temperature Strip Tenka terrorist Testing Texas BBQ Company Text Thai Thanksgiving in IT The Mooring Thomas Thought Leader Three Gorges III TIA 942 Timesheets Toby Keith Toddlers traditions Transition treehouse turnover TV Twitter unspoken moments Valentine's Day Value Vendor Venezuelan Verizon Vermont Video Vietnamese voice VoIP Watertown Wedding Westborough Korean Restaurant Westborough MA. StormCam WiFI Wi-Fi Wilbraham Wine Worcester work work life balance working Yama Zakura Zem Han Zitis

Entries in Crisis (2)

Monday
Oct082012

When IT is in Crisis

We often get asked to help companies return their environment to stability.  Companies can erode customer good will and lose business with extended or recurring IT issues.

When IT organizations find themselves in crisis, digging out can be difficult. 

We’ve covered stability issues in the past, in two posts “The Insanity Must Stop” and “Change Management.”  For a company to reach out for help indicates strength.

Why strength?

Weak organizations will continue to try an solve the issue.  When “really bad,” this often takes a toll on staff.  People make mistakes when they are tired.  Engaged management sets ego aside and asks for (ultra-confidential) assistance.

A strong organization will recognize a fresh perspective is often needed to objectively understand the situation and make positive strides.

We deploy a team of experienced professionals (20+ years) with interdisciplinary perspectives, including:

  • Process Analysis – to review processes for issues, particularly in the change and configuration management space (often weak change processes can cause instability.)
  • Data Analysis – to gain a perspective from any ticketing system (such as BMC Remedy, ServiceNow, CA Service Desk Manager) or error logs.
  • Interviewing – To collect data from the front lines. 
  • Subject Matter Expert – if the client appears to be having issues in a single technical discipline, we bring a Subject Matter Expert in this discipline.

We do a time boxed assessment, limiting ourselves for a first pass.  We use our methodology of SCAN-FOCUS-ACT-CONFIRM:

  • Scan – Rapid, high-level, holistic assessment identifying areas for further research. During the scan phase, conceptual models are established, risks and opportunities are identified
  • Focus – Targeted analysis detailing opportunity and risk areas. Areas are scrutinized and evaluated thoroughly.
  • Act – Execute action plans delivering solid results using client’s staff, vendors, and third-party consultants.
  • Confirm – Assure deliverables achieved the desired business impact, reentering the process if needed.

The S-FAC methodology originated with MG Taylor’s Scan-Focus-Act model with a goal of a swift assessment, followed by a focused effort and action. Harvard Partners added the Confirm step as we believe validating results achieving the intended business goal is critical.

In a production stability assessment, we do a quick SCAN developing a list of FOCUS areas for further investigation.  We do this in a week to two weeks (of really long, completely intense days.)

At the end, we are able to produce an assessment with actionable steps to be “ACTED” on by the company, either independently or with our ongoing participation.

Can companies do this analysis on their own?  Of course, and we find companies doing it every day.  When companies need a fresh perspective to get them over the hump, external views are often vital.

Can companies spend more time on analysis?  Yes.  Lots longer.  We simply believe in staunching the bleeding quickly, and then taking more time for further analysis.

What thorny IT issues have you experienced?

Thursday
Nov122009

Information Technology Crisis Management

“I don’t believe in Disaster Recovery.”

What a pretty bold statement in an initial interview. The interviewer/hiring manager was a former executive of a major “hot site” company, who played a pivotal role for many clients during the World Trade Center bombing of 1993. He had a fiery reputation, and visibly stiffened at the comment.

My second sentence put him at ease, reinforced our value set, made us friends for life, and most importantly got me the job!

“Preparing for the ‘big one’ solely can leave you ill prepared for the myriad of daily events nipping at us every day. Prepare for the daily events, and use the same process for initiating processes for the ‘big’ events.”

While firms spend millions on high availability, redundancies, balancing, geographic diversity, N+1, etc., inevitably “perfect storms” lead to user impacting outages. How IT staffs respond in a crisis can differentiate companies and impact the bottom line.

The key is having everyone understand their goals, roles and responsibilities in a crisis. And during a crisis, there may be a change in an individual’s role as needs warrant.

For example, once an issue escalates to a crisis it’s useful to have at minimum two communications outlets. There needs to be management discussions focusing on impacts, alternatives, communication (to users, clients, regulators) while the technicians focus on eliminating impacts and understanding root cause.

Who is involved in each grouping is best determined in the calmness of an outage free period.

Management groupings will often include the CIO, Human Resources, Security, Public Relations, Business Continuity and the impacted business area. Technical groupings will include representatives of all the major functions, with one area tracking all the activity and logging for subsequent analysis. Frequently, the techies establish their own information communications using instant messaging or Twitter (used in a publish-subscribe manner.)

When a crisis is apparent, the notification and escalation processes have to be well honed. We often see organizations operating on predetermined voice “bridges”, with notification via text, email, phone call (in many organizations this is via an automated notification system).

There needs to be a protocol for entering the bridge, doing roll call, and general etiquette around bridge participation.

These processes are best developed in advance of need and rehearsed with staff in table top exercises.

When Information Technology leads the way in developing low overhead crisis response, and orchestrates rehearsals reducing the processes to practice, preparedness for managing the “big one” is well underway.