Looking for something? Look here!
Want more unvarnished truth?
What you're saying...
What I'm saying now
I think tag clouds are pretty, and not to be taken overly seriously
##MoveWithGary #Home Inspection #MoveWithGary 111 Chop House 75 on Liberty Wharf 9/11 A Broth of a Boy ABCs Abiouness accountability activities alcohol Allora Ristorante Analysis Angry Hams ANSI/TIA 942 Anthony's Pier 4 Apple Application Armsby Abbey Arsenal Arturo's Ristorante Ashland AT&T Audio Automation baby Baby Monitor babysitting Back To School Bad News Bangkok Thai banks lending movewithgary Bar Bay State Common baystateparent BBQ BCP Bees BeeZers Before I die I want to... behavior Big Bang Bike Bill of Rights Bistro Black Box BlackBerry Boston Boston Marathon boundaries Boyston BPO brand Breakfast Bridge Bring Your Own Technology Budget Burlington Burn Burrito buyer BYOD Cabling Cambridge Camp Campaign career Casey's Diner Castle casual cCabling Cell Phone Central Square Change Management Cheers Chef Sun ChengDu Chet's Diner Children Chinese Christmas Christmas Families Holiday CIO Cloud coddle collage College College Acceptance co-lo Co-Location Co-Location Tier Power Cooling Comfort Food Control Country Country Kettle Crisis customer dad Dad Phrases damage daredevil Data Center Data Center Design Davios Day Care Dead Death declaration Del Frisco's Design Desktop Video dinner Disaster Recovery Divorce Do Epic Shit dodgeball downsizing Downtown Crossing DR driving Droid Easter Economic Kids Edaville Education Elbow Night Elevator Employee Engagement Erin Estate Planning Etiquette Evaluation events Exchange Expiration Dates Facebook Failing family Family Law Fatherhood Favorite things first time buyer Flash Flemings Fogo de Chão Food Hits and Misses Format Foundry on Elm Foxborough Frameworks fraternity Fraud French Fried Clams friends fun Fusion Generations germs Girl Scouts girls Global Go/No Go GPS Grafton Grandchild Grandpa Harry's hazing Healthcare Healthy Choices while Dining Out Help Desk Hisa Japanese Cuisine Historic holiday Home Home Inspection hope Horizons hose Hot Dog Hurricane IIT Assessment incident Indecision Indian Infrastructure Inn Innovation Insurance Internet Inventory Management iPhone IT IT Assessment IT Satisfaction Italian Jack Daniels Jakes Restaurant Janet Japanese Jazz Joey's Bar and Grill JP's Khatta Mitha kickball kids Laid off Lakes Region Lala Java Leadership Learning legacy Legal Legal Harborside Les Zygomates L'Espalier Liberty Wharf life transition lights out Linguine's loss Love Lucky's Cafe luxury luxury home M&M Macys Thanksgiving Day Parade mai tai Managed Application Services Managed Services managers Mandarin Manners Mark Fidrych marlborough marriage Mary Chung mass save Maxwell-Silverman Mediterranean meetings Memorial Day memory Mendon Mergers Mexican MiFi Migration Ming III miss MIT MIT CIO Symposium mmortgage Mobility Moes Hot Dog Truck MOM money mortgage Mother MoveWithGary Moving on Name nature neanderthal neighborhood Network new listing New York Marathon newborn Northborough Not Your Average Joe's Nuovo Nursing On-Call Operations Operators Oregon Club Organization Pancakes Pandemic Parental Control Parenting Patch Peeves People Perserverance UMASS growth Photography Play Plug and Run Predictable Pride Problem Process Production program Project Management propane PTA. PTO PUE QR Quick Response Rant re/max Real Estate Realtor Recognition Red Rock Resiliency Respect restaurant Restaurant Guy RFP ribs Ritual Root Cause Analysis rReal Estate Sam Adams Sandy Sapporo savings School Sea Dog Brewing Company Sea Dog Steak and Ale Seafood Seaport Security Sel de la Terra Service Service Desk Service Indicator Light sharing ShearTransformation SHIRO Shit Pump Shriners SHTF Simplification Skunk Works Skype Sleep sleepovers Sloan Smith & Wollensky soccer Son SOP sorority spanking Squarespace staffing staging Starbucks Status Reporting Steak Steve Jobs Storage Strategy stress Summer Sushi swimming Tacos Acalpulco teacher Technology Teen Telephony Temperature Strip Tenka terrorist Testing Texas BBQ Company Text Thai Thanksgiving in IT The Mooring Thomas Thought Leader Three Gorges III TIA 942 Timesheets Toby Keith Toddlers traditions Transition treehouse turnover TV Twitter unspoken moments Valentine's Day Value Vendor Venezuelan Verizon Vermont Video Vietnamese voice VoIP Watertown Wedding Westborough Korean Restaurant Westborough MA. StormCam WiFI Wi-Fi Wilbraham Wine Worcester work work life balance working Yama Zakura Zem Han Zitis

Delivering Bad News

To:     All Personnel

From:     Information Technologies

Subject:     System Outage

 As Infrastructure & Operations professionals we all delivered these emails or memos to our users. It pains us when we have to communicate this information, but how and what we communicate ultimately defines the respect we receive from our users and senior management.

 I am reminded of an outage that occurred while I was at a seminar in New York City. Our trading system was impacted while data was being replicated to our Disaster Recovery systems. We could not assess the impact of failing over, so we attempted to fix the problem. I was immediately paged and alerted the CIO. It appeared we would not be able to restore the trading system in time for the start of day so I immediately packed and got the 6:30AM flight out of New York. On my way to the airport I informed the team I would speak to the CEO as soon as I landed. I got on the plane (still on the same conference call from 3:00AM), and who do I see 10 rows in front of me, but the CEO. I couldn’t get to him on the flight so I figured I would catch up to him in the airport. Luckily as I was running through the terminal I was informed all systems were operational and business would proceed as usual. I caught the CEO and, relieved from the news I just heard, joked how it was funny we were on the same flight.

Unfortunately, situations do not always end up this way, and you must find ways to communicate bad news to your users. Over the years, the following guidelines served me well:

  • Determine the sender of the message – The sender of the message will also be the person to receive any questions or comments about the incident and status. A message sent from a person, instead of a generic mailbox, will carry more credibility, but will yield more questions.

  • State the problem in the first paragraph – Let people know what happened in the first sentence. Use the second sentence to communicate business impact. Never use any technical terminology and never describe systems by their internal IT name. Always describe systems by the business functions hosted on the system.

  • Identify functional business systems – The first paragraph notifies users as to systems not available. The second paragraph lets them know systems that are available. Users want to know if they can do their job and your communication must not be ambiguous in this context.

  • Give estimated time for recovery or time of next status message – People want to feel informed and in some control. Frequent status updates helps to achieve this goal. If an estimated time of recovery is unavailable, let people know when the next status notification will be issued. Over time, you’ll be able to either have accurate estimates, or will get a “gut-feel” for the length of outages.

  • Designate a point of contact – Users will have questions. Give them contact information. The Help Desk would be our first choice. Questions will also be directed at Desktop Support personnel as they have the most frequent contact with users. Do not forget to notify them before you notify your users.

  • Do not apologize if it was not an internal problem – Sometimes, third-party hardware and software fail. We try and prevent this from happening, but there are times when it is beyond our control. Only apologize when you have something to apologize for.

As problems become more complex and require larger numbers of IT personnel, we recommend using a dedicated Problem Communication Manager for generating both internal IT and external updates. This will simplify the job of the Incident Manager and provide better service to users.

Defining problem communication processes, people, and templates delivers a higher level of service. Users will appreciate the communication and it will be one less task for IT personnel involved with the problem.


Writing Performance Appraisals

I hate writing performance appraisals, or reviews.

In my mind, people sit down twice a year and formally document staff performance. Inexperienced managers often write “nice” reviews, with little constructive feedback. More senior managers often skip writing the performance appraisal altogether, again limiting feedback. Reviews in many ways give an objective appearance to a subjective process.

My personal dislike of reviews has to do with the fact they are generally documenting an extended period. My preference is to have an ongoing daily/weekly dialog, with the review capturing a snapshot of those discussions over an extended period.

When writing a review, I like to start by

having the reviewed submit a self appraisal. Most people are tougher on themselves than you may think, and I find it good to capture what the individual is thinking.

It’s nearly impossible for me to write reviews during the work week. There are simply too many distractions. Reviews are reserved for weekends, in the office, with the music loud.

To pull together the document, I need the:

  • Objectives – objectives are developed at the beginning of the review period. They should be SMART – Specific, Measurable, Attainable, Realistic and Trackable. Be aware objectives evolve over time, and your review needs to reflect this

  • Personal 1:1 notes – capturing discussions over the review period

  • Self Appraisal – The reviewees self appraisal

  • Prior Reviews – checking for any prior “messaging”

Your company probably has a format for the review. You’ll need to follow the company format. IN general, you’ll want to:

  • Review objectives achievement - thoughtfully review each objective and the commitments. Be cognizant of any evolution in the objectives (hopefully this is captured). I try to be as objective as I can, recognizing it’s easy to make this totally objective while the better result often lies in subjective analysis

  • Comment on the individual’s progress against a series of skills competencies – if your company doesn’t have these, I suggest using a tool (such as http://www.performancereview.com). One benefit of a tool approach is the commentary around each of the competencies. It’s fast to pull together this section of the review on a consistent basis

  • Write a summary – The summary is often the place where the entire review is captured (hence – summary). Personally, I use a structure of:

    • Opening sentence capturing the review

    • Specific examples of good competency achievement

    • Specific examples of where competencies need improvement

    • Thoughts around related objectives/education for the following review period

With the review written, sleep on it. Give the benefit of a day or two to evolve your thinking. Go back, and edit the review with the benefit of time. Make sure your messages are clear; for a particularly challenging review, have your boss and/or HR review the review.

When it comes to presenting the review, hopefully the messages are very direct and not a surprise. You certainly don’t want the review meeting being the first time someone is hearing the content.

Set aside a time and location for the meeting. I like using a conference room or some other neutral place. I give the person their review, and leave the room for ten minutes to give them time to review (unless the messaging might incite someone, such as a extremely rare review ending in termination.)

Upon return, I then go through the entire review top to bottom paraphrasing each section. To me, the discussion is the most important part of the review. Make sure it is a dialog and not a monolog in presentation.

When finished, I always ask:

  • Did I capture the essence of you? What didn’t I capture? – If I missed something in preparing the review, let’s discuss now.

  • How am I doing? What can I do to better support you? – Always keeping the questioning on how to better the reviewed.

This summarizes writing a performance appraisal in a very short form. Multiple day classes are offered detailing the subtleties around reviews. My biggest message to you is to:

  • Give honest feedback on an ongoing basis – NO SURPRISES

  • Take the time to reflect balanced feedback in the review

  • Make sure the discussion is well rounded.

You are there to help lead your staff, and the performance review is one tool as your disposal.


Morning Operations Meeting

“Nothing productive ever happened in a meeting,” a friend once stated. He is a thoughtful guy, and his comment was not one to be idly dismissed. As you ponder this during the next meeting you attend, consider the value of a daily touch base on operational issues.

DAILY? Surely you jest.

Whether in crisis or not, a daily session is imperative in any well run operations area. And believe it or not, the meeting can be accomplished in under 10 minutes! It’s all about predictability and preparation.


When running meetings like this, use a conference bridge with the same ID each day. Attendees shouldn’t have to search around for the contact information. Use an acronym if you can (the Morning Operations Meeting can be referenced as MOM. A conference bridge of CALLMOM (2255666) is easy to remember.

If there’s a critical mass of people at one location, try to use a conference room at that location to run the meeting. Far flung attendees participating by conference bridge is one thing, “locals” can come attend the meeting (rather than sitting at their desks reading emails!)

Pick a time when everyone can attend, based on your business day. Financial services companies will want to have the meeting well before the US stock market opens at 9:30AM (8:00 AM is a good time). If you are a retailer with stores opening at 8:00AM, an earlier time may be more appropriate.

Start the meeting on time each day. Nothing ruins the attendance and contributes to time creep than a meeting where the start time waffles. To do this, a backup chairperson should be in place to start the meeting if the chair is delayed.

The meeting should have the same agenda each day:

  • Roll call

  • Area by area review of any major (customer impacting) issues over the past 24 hours, with an emphasis on any active issues

  • Follow up on prior action items

Minutes should be captured, and emailed to each of the areas.


Preparation is another key to this meeting. Since the agenda is the same each day, the “areas” for review can be pre-populated on draft email. Over the 24 hours from the last meeting, Operations and the Help Desk should “contribute” major customer impacting issues to the draft. So when the meeting actually happens, the Chair is following a script of the meeting (literally reviewing a draft of the “minutes”.)

As the meeting is held, the chair can “prompt” speakers if certain issues are glossed over or missed. In this manner, major issues are not missed.

Details are not covered in this status meeting. If the issue is still active, it is placed on “follow up,” and brought back to the meeting. The chair has discretion for cutting off a discussion.

Once the meeting is completed, a brief Summary should be added to the email (suitable for reading on a BlackBerry) and the send key pressed. A wiki can also be used for this.

With predictability and preparation, the meeting will flow smoothly. Plan the meeting will run long the first week or so as people adapt to the meeting style.

Once the minutes start being read, it’s common for people to start wanting the “edit” the minutes after the fact. Some will want immediate retractions issued. My recommendation is to offer to add a “correction” section at the bottom of the minutes and issue as a part of the daily cycle. Do not get into multiple MOM minutes.

Savvy areas will want to review the “script” in advance. Why not? It allows the overall product to be stronger provided the information is factual.

And one last fun suggestion. Play into the MOM (as Mother) theme. “It’s OK to tell MOM anything. MOM is here to help.” It allows a subtle mindset shift.

And remember, you can fool some of the people all of the time, all of the people some of the time, but you can’t fool MOM.


Project Planning 101

A number of years ago, my 11 year-old son came to me and asked me to sign his homework. Being an engaged parent, I decided to actually look at his work. To my surprise, he had been asked by his teacher to complete a project plan for his upcoming math project.

I looked at his paper and realized it asked for the following:

  • A list of materials (“resources”)

  • Steps to complete the project (“tasks”)

  • A draft drawing (“milestone”)

  • Due date (“deliverable”)

  • His name (“project manager”)

  • Parent signature (“signoff”)

I immediately asked when he found the time to go to Microsoft Project training

and how he managed to connect to Project Server so he could allocate enterprise resources to his project plan. He turned to me and asked whether I had been “inhaling” (this was when Bill Clinton was running for president) and then proceeded to tell me that he simply wrote down what he needed to complete his math homework. When asked how long it took, he told me 5 or 10 minutes, but that included keeping his lines straight.

While I poke fun, I wanted to share the importance of starting simple. Many technical managers become overwhelmed when asked to do project plan. They don’t understand how to use Microsoft Project and believe it is the key to building a project plan. Understanding your project and articulating what needs to be done (tasks) and the people required (resources) are the building blocks to a robust project plan. These can be documented in a Word document or and Excel spreadsheet. Think about logical sections of your project and you will start to develop phases. Each phase should have at least one deliverable. Document this deliverable and it becomes a milestone. Printing these documents and drawing lines between tasks and phases will create dependencies. Now you have what you need to use a project management tool, such as Microsoft Project.

People believe project management tools are all about building project plans. They force a their project plan into the tool, rather than using the tools to help them predict deliverable dates, resource levels, conflicts, budget issues, and project risks. A well designed and implemented project plan is a living document used to clarify project roles and responsibilities for the project manager, participants, and management. If done correctly, it can save time and provide real-time views of project status.

So, when starting any project, take the time to create a simple plan. Be comfortable and confident in the tools used to create the plan. Remember, the content is much more important than the presentation. If the tool you have selected is not a project management tool, then spend the time insuring your plan is comprehensive and robust before transferring the data to a project management tool. Use the project management tool to help you estimate key project metrics, and you will be successful.


Mergers and Acquisitions

“This is a good old fashioned strategic merger.” I heard those words from the Chairman of the Board in 1990 while working for an office products company. 20 years later, this company’s headquarters building is condominiums.

Companies they had “acquired” over the years heard the same thing…and suffered similar fates.

So while the trends I’ve observed in nearly every combination may not be faithfully generalized in all cases, they are a synopsis of what I’ve personally experienced.

Kelley’s Humorous Laws of Mergers

  1. Mergers are a legal/tax framework. There are only acquisitions.

  2. The acquiring company is the “winner.” Their self image is one of a brilliant staff, and they do many things correctly.

  3. The acquired company is replete with cost savings opportunities. The staff is expendable, and made many errors leading to the company’s demise.

  4. “We will examine all systems and chose the ‘best’.” This is management-speak for saying we will choose one infrastructure and migrate everyone to it. It would take too long to interface disparate systems for the perceived functionality gains.

  5. Some companies let acquisitions run autonomously. This is typically until the senior execs or family selling the company retire and is solely meant to placate the prior ownership.

  6. Companies acquiring aggressively are prime acquisition candidates. Areas not fitting strategically will be divested. “Management buy outs” indicate a non-strategic area, or an area the acquiring company didn’t choose to understand or develop.

  7. Divergent company cultures are often the most challenging areas in an acquisition. Company cultures often reflect the stereotypes of the geographic region.

  8. The first system deployed to newly acquired companies is often expense reporting.

If you work in a recently acquired company, be positive and upbeat and try to make the new organization work. Keep your eyes open, and prepare for a possible plan B (if only updating your resume.)

You could read into this I am against acquisitions. I’m not; the office products manufacturer referenced in the story open was facing a rapidly consolidating retail market (birth of the office products superstore) and wouldn’t have otherwise survived. The acquisition allowed staff the opportunity to gracefully find opportunities rather than suffering through a company agonizingly ceasing operations. Change is inevitable, and an acquisition is simply another change.


Cutting the IT Budget

We need to cut the budget,” the CFO pronounced. “Things are tough out there, and we need to trim now to fight another day.”

The truth is most budgets have been tightened for a long time, and all CFOs are playing it conservatively.

“We used to be lean and mean,” grumbles an IT Director. “Now we’re just mean.”

It doesn’t need to be that way.

You can make a budgetary exercise a bit of a game, and rather than make it a loser sport, make it fun. Award the group or person with the largest budgetary percentage savings. The “award” can be a simple certificate (as opposed to an all expenses paid trip!)

Assuming you’ve already trimmed subscriptions, office supplies and other line items representing a small overall percentage, you’re probably looking at four major areas for further cuts.

  • Staffing – often, staffing is the first metric CFOs look at because they relate well to it and for years we’ve been saying the other line items are not able to be cut! Nobody likes to cut staffing, and as a reality you may have to do so. Some methods you may want to use include releasing marginal contributors (who in many cases know they are marginal and welcome the “package,”) and bringing in outside contractors.
    Bringing in contractors? This seems counterintuitive. The truth is there’s pressure on rates, and flexible staffing models allow you to turn up and turn down the spend rate. Some companies furlough contractors the last couple weeks of each year, rather than paying contractors to be around during a lighter work period (other companies use the contractors as a way to give permanent staff a year end break.)

    While often a challenge in larger companies, cutting base pay or bonuses (if bonuses are still given) is often something many employees would prefer to laying off their friends. You’ll have to explore this with Human Resources, and be crystal clear on communicating to the staff.

  • Software Licensing / Equipment licensing – Do your homework and make sure you are current and up to date on your inventories. Often companies find they’ve exceeded a license agreement, and some suppliers are willing to package an adjustment in with a purchase (while others resort to sales by extortion remediation!) In some cases, companies oversubscribe to particular software, and may be able to reduce license counts commensurately. Don’t fall into the trap of signing long term agreements to address…what feels good today will bite you in a couple years.

  • Maintenance – This is suppliers most protected line item! Major suppliers invest heavily in protecting their installed “annuity” revenue base. It is often well worth time exploring the maintenance agreements on major vendors, and dissecting them into component parts.
    Often there’s an upcharge for 24x7 service. Do you really need it?

    For example, a modest branch office may not need 24x7 hardware support for a router, especially if redundancy is in place. In fact, one financial services firm has procured (older generation) replacement routers for branch offices and has the desktop support area do “swaps”. The branch office ultimately gets better service, and the failed equipment is repaired on a time and materials basis.

    The same holds true on personal computers or phone devices. Why have a support contract at all? The machines continue falling in price, and a replacement is often able to be installed for a fraction of the cost (overall) of a maintenance program. We’d suggest the phone switch get premium service, and not the phone device.

    We’re not suggesting dropping maintenance across the board. Look at each item of maintenance and determine if there are other creative ways of dealing with it.

  • Communications - voice and data charges are another large line item in most budgets worthy of inspection. Many steer clear of this as they believe they really don’t control it (i.e. I don’t make all those calls) and the contracts already in place have a commitment period.
    Start with an accurate inventory, making sure all the line items are still in use (this is a tedious task, and some consulting firms will do this work for a percentage of the savings.)

    Are you making the most of the technology you have in place? For example, a call to a branch office “on net” is often cheaper if routed over the data network (especially if you have global offices.) This requires your telephony and data communications teams to work together, making sure the data network and phone systems are configured to eliminate dropped calls/ echo/ busies and the like. Note: while a coordinated dialing plan is a huge convenience, it is not needed to make this cost savings leap.

    Is your company still paying for cell phones? It’s not uncommon for companies to install the infrastructure for email access (like a BlackBerry Enterprise Server) and have staff fund their own devices.

    Issue an RFP for the remaining services. Communications contracts are often for multiple years, and existing suppliers will be reticent to adjust pricing if they feel you are locked in. Look at your contract – many have modest minimums, so you can actually switch vendors without violating your existing contract! Your incumbent suppliers need to believe their business with you is at risk for you to get breakthrough pricing. This requires a bit of hardball, and staff need to be echoing the same message to the vendors.

Then again, there is another less “in your face” approach we saw successfully used as a major financial service firm. Each (major) vendor was contacted and told a 7% reduction was requested. While there was some hesitation, every vendor came through with a 7% reduction. One might argue a larger reduction percentage would have yielded greater savings overall, and we’d counter vendors understood the need for a modest cut and they were willing to participate rather than risk their business. In many cases, the vendors ultimately appreciated the soft approach rather than the stick.

One last word on the subject of cutting budgets. Please keep your training budgets intact. Sending staff to training is like changing the oil on the car….it must be done or you’ll have problems later. Cut travel budgets, encourage more cost effective hotels, explore online courses….and keep your staff knowledgeable, up to data and engaged!



“All groups will be dressing for Halloween,” announced the company President. We were an “east coast” company recently acquired by a “west coast” firm, and the new owners were mandating participation. “Don’t question it, just get on with it,” was the response as the senior team started reacting.

The IT staff had similar opinions. “We’ve got real work to do,” “I won’t participate on religious grounds,” and “I’m diabetic” were some of the comments.

“We have two ways to do this. We can either limp along with it, or we can own it and go way over the top,” was my response. If we’re going to do something, let’s do it well.
The light bulbs started going on throughout the room.

“We can go over the top?”

“Don’t get me fired.”

So the team went to work. There were a bunch of us attempting to pick up the game of golf, so each person implemented a “horror golf hole” in their workspace….even the non-golfers.

Everyone tried to outdo the others. One of my favorite “holes” was a bed of nails…the ball went into the top, and bounced around to a series of predetermined “strokes”. The spouse of this team member is a professional carpenter, and the “props” were first rate.

Halloween came, and no IT work got accomplished. Everyone in IT had to check out each other’s golf hole and costume. Then word got around the building, and the rest of the company had to come down and check out each team member’s golf hole. Then the President gave out awards, and it was no contest. The IT staff carried the day.

Afterwards, the IT staff milled about recapping the day.

“This was a great day. Everyone showed their great potential and talents. And you know, the new owners are pretty cool,” was my summary. Everyone agreed, “We’ve got a year to plan for next Halloween.”

Halloweens came and went, with none topping the first one. Years later, this team stays in touch with each other unlike any other team. Taking an extreme view, what could have been an exercise in corporate compliance became a great team building exercise. You have to let the team establish relationships beyond simply the work environment.

Me? My golf hole was a simple uphill across a river of red dyed water (blood?) I also learned how frightening a chain saw (without chain, of course) sounds in an office building!


Data Center Disciplines

I have been teased my entire career about my nearly obsessive behavior around keeping data center rooms neat and tidy. While I’d love to blame my Mother for my neatness, the truth is keeping a data center clean is about one word: discipline.

Having a neat and tidy data center environment sends a reinforcing message to everyone entering about the gravity of the work performed by the systems in the area. This is important for staff, vendors, and clients.

The observational characteristics I look at when I walk in a data center are:

  • Life Safety – are the aisles generally clear? Are there Emergency Power Off switches and fire extinguishers by the main doors, is there a fire suppression system in place, is the lighting all working….

  • Cleanliness – Forty years ago data centers were kept spotless to prevent disk failures. A speck of dust might make a hard drive disk head fail. These days, disks are generally sealed, and can operate in pretty rough environments (consider the abuse of a laptop disk drive.)
    While disk drives are generally sealed, why should data centers be dirty? Look for dust, dirty floors, and filthy areas under raised flooring. One data center I went in had pallets of equipment stored in the space…was the data center for computing or warehousing?

  • Underfloor areas – are the underfloor areas, assuming use as an HVAC plenum, generally unobstructed? More than one data center I’ve been in had so much cable (much abandoned in place) under the floor the floor tiles wouldn’t lay flat. This impacts airflow, and makes maintenance a challenge.
    I also like to see if the floor tiles are all in place, and if some mechanism is used to prevent cold air escaping through any penetrations. 30% of the cost of running a data center is in the cooling, and making sure the cooling is getting where it needs to be is key. (While at the opposite end of the space, I like to see all ceiling tiles in place. Why cool the area above the ceiling?)

  • HVAC – are the HVAC units working properly? Go in enough data centers, and you’ll learn how to hear if a bearing is failing, or observe if the HVAC filters are not in place. As you walk the room, you can simply feel whether there are hot spots or cold spots. Many units have on board temperature and humidity gauges – are the units running in an acceptable range?

  • Power Distribution Units – are the PDUs filled to the brim, or is available space available? Are blanks inserted into removed breaker positions, or are their “open holes” to the power. When on-board metering is available, are the different phases running within a small tolerance of each other? If not, outages can occur when hot legs trip.

  • Hot Aisle/Cold Aisle – Years ago all equipment in data centers was lined up like soldiers. This led to all equipment in the front of the room being cool, and all the heat cascading to the rear of the room. Most servers today will operate as high as 90 degrees before they shut themselves down or fry. By having a hot aisle/cold aisle orientation, including blanks in empty shelves on servers, cooling is most effectively in place. Some organizations have moved to cooling being in the racks as a designed alternative.

  • Cable plant – the power and communications cable plants are always an interesting tell tale sign of data center disciplines. Cables should always be run with 90 degree turns (no transcontinental cable runs, no need for “cable stretching”). Different layers of cables under a raised floor are common (power near the floor, followed by copper communications then fiber). (A pet peeve of mine in looking at the cable plant is how much of the data center space is occupied with cables. Cables need to get to the equipment, but the cable plant can be outside the cooled footprint of the data centers. Taking up valuable data center space for patch panels seems wasteful. One data center devoted 25% of the raised floor space for cable patch panels. All this could have been in not conditioned space.)

  • Error lights – As you walk around the data center, look to see what error lights are illuminated. Servers are often monitored electronically, and error lights utility is lessened is a argument. That said, error lights on servers, disk units, communications units, HVAC, Power Distribution units and the like are just that: errors. The root cause of the error should be eliminated.

  • Leave Behinds – what’s left in the data center is often an interesting archeological study. While most documentation is available on line, manuals from systems long since retired are often found in the high priced air and humidity controlled data center environment. Tools from completed projects laying around are a sign thoughtfulness isn’t in place for technicians (I’ll bet their own tools are where they belong).

  • Security – data centers should be locked, and the doors should be kept closed. Access should be severely limited to individuals with Change or Incident tickets. This helps eliminate the honest mistakes.

While far from an inclusive list, this article is to help silence my lifelong critics about my data center obsessions. These are simple things anyone can do to form a point of view on data center disciplines. Obviously follow ons with reporting, staff discussions, etc. is appropriate.


Alcohol as a Truth Serum

Alcohol has a strange way of impacting people. For some, it is a truth serum reducing inhibitions around what NOT to say. Others end up weaving great stories when imbibing on some spirits.

Such was not the case when Cindy’s husband approached me during a holiday party. “I don’t like it when Cindy gets paged. She doesn’t get paid for it and it interrupts our activities.”

This was NOT what I wanted to hear.

First, I really dislike Holiday Parties. Tending to the introverted like so many IT types, I’ve conditioned myself to get out of the corner and make a pass through the entire event. Once I complete my tour, I reward myself with a “get out of party” card. Second, having a discussion with a concerned spouse during an event can be an opportunity or a disaster.

Cindy, the Production Control Scheduler, looked on in horror. Her husband is a big burly man, intimidating at first sight. He was raising a concern, one I suspect Cindy had heard about privately on other occasions.

“I don’t like it when Cindy gets paged, either. It’s a bother for our Operators, too. What it often means is something is wrong in the schedule Cindy produced and by giving her direct insight to the issues we place her in a position of addressing them permanently so she won’t get paged again in the future.” Hubby suddenly began to see the light around the accountability Cindy had, and her direct power to impact the results.

It’s important to note we both worked in a manufacturing company having the rules of a Union without the Union…this played into my continuation. “Cindy is entitled to 4 hours pay every time she is paged. I know Cindy feels awkward about putting in for 4 hours if it was a quick call from the operators. I believe she only records “serious” time spent on issues, and on Monday I can sit with Cindy and review our reimbursement processes.”

At this point, Hubby seemed to be more interested in Cindy’s time recording practices than anything else, and I suspect Cindy had further conversations with Hubby in the coming days.

There are couple areas where conversation is good with the subject of on-call.

Companies need to be clear on their on-call approaches and whether any financial or other remuneration is received. My own sense is an occasional quick call comes with the territory. If logging on and researching is needed, we need to acknowledge the impact. Personally, I’m not a fan of rigorous time reporting on these kinds of interruptions. I’d rather be more lenient of someone leaving early to play golf or see the kids play (or whatever the passion is), or taking an occasional day off. (As a manager, I believe such accommodations should be within a week or two of a significant on-call event. I really dislike being confronted with 42 days off accumulated over the last two years.)

Staffs need to make their friends and family aware of what on-call means for them. Few industries have professionals with 24x7 requirements of IT. While “smart hands” may be in a rotation, managers are often called into every major issue and are on conference calls a large part.

Staffs also need to think about the root cause of the issues. Breathing a collective sigh of relief after a remediation is just the first step. Thoughtful analysis of the root case for the issue needs to be performed with an eye towards learning how to prevent issues in the first place. This effort must be focused on improvement, not on a witch hunt. “Post Action Review” is an important concept, where individuals and their managers can present findings to senior IT managers where a lively, thoughtful discussion can take place.

Through thoughtful, reasoned communications improvements can be made reducing issues and outages.


Wanted: Technology to Drive Process

“Technology driving process; It’s not supposed to work that way.”

Anyone with formal training in process engineering knows you start with defining and optimizing your processes, and then use technology to streamline those processes. In an ideal world, this works perfectly. Layer organizational structures, system ownership and governance models, and personalities, and people naturally gravitate towards what they know best; technology.

A number of years ago I began moving my Infrastructure & Operations division to more of a process-based organization. I, and my management team, attended a series of seminars given by the late Dr. Michael Hammer. We received our certificates in “Process Mastery” and, with our newfound evangelical powers, were ready to transform the organization.

Barbara, having a reputation as an overachiever, volunteered her group, the Desktop Support, Engineering, and Help Desk department, as the first to make the move into the world of process. Over the period of a year, Barbara documented existing processes, designed new process where others were missing, created a process map, developed Service Level Agreements with users and other IT groups, and conducted training sessions with her team. The results were better delegation of tasks to the right individuals; managing to metrics, happier employees, and, most importantly, improved service levels.

As Barbara’s manager, I pushed for more improvement. Barbara responded by identifying Help Desk requests that could not be resolved on the first call and required assistance from others in the IT organization. In reviewing the list, Barbara realized 30% of the tasks could be shifted to the Help Desk and drive down costs while dramatically improving resolution time for the user. Requests such as resetting passwords, granting access to network file shares, provisioning user logins, email accounts, and printers, distribution of remote access security tokens and instructions, and creation of new user profiles were all currently being performed by senior systems administrators. The management team thought Barbara’s idea was great, and she was empowered to make it happen.

Barbara thought this would be simple. She had achieved what she thought was buy-in from the entire infrastructure and operations organization. What she didn’t expect was resistance around what people believed gave them power. Employees in other groups were fine giving over these mundane tasks so long as they still had control over approving each transaction. They felt this authority (and the trust that went along with it) is what made them special to me and others in the IT organization. What they failed to see was the erosion in the level of respect they received when they complained about not having enough resources, but failed to seize this opportunity.

The solution came through implementing technologies enabling Help Desk personnel to grant user privileges without being server administrators. Once the systems administrators saw they had not lost any control, were able to delegate tasks to the Help Desk, and were able to focus on more high value projects, they began to think about methods to offload other processes to the Help Desk. Success was achieved.

The lesson from this story is the need to discover what drives people before reengineering their processes. In this situation, the sense of control the ability to manage the technology was the drivers for the Systems Administrators. Empowering the Systems Administrators to use technology and enable the transferring of some of their processes made them supporters and eventually advocates.

In the case of the Systems Administrators, they defined their processes by technology. Therefore, technology became their driver for re-engineering their processes.

Some of the “squishier” skills, such as process, project management, client management, and budget management can be difficult for administrators and operators to understand or appreciate, and can be in conflict with their priorities. In a production IT shop, keeping systems up and running and eliminating any user downtime is the top priority. Asking people to take time away from their priorities, particularly in these challenging times, may be counter-productive and may produce defensive behavior. Take care in understanding your audience .