IT Operations – The Unsung Heroes

This is a story of how one company and its operations staff kept the lights on in the face of post-Hurricane in New York City.

Three blogging websites are under my general control:

• Garylkelley.com – a site about technology, fatherhood and restaurant reviews
• Curriculotta.com – an “alter ego” site for the properly starched shirts over at Harvard Partners
• Markfidrychfoundation.org – a site for furthering the community work of the late ball player Mark Fidrych.

Squarespace is used to “host” these sites.  WordPress was our original choice, and served us for a while.  I was just never a fan for how WordPress “thinks.”  Personally, I prefer Squarespace.

Squarespace uses PEER1 as their co-location provider, located in New York at 75 Broad Street.

You can imagine my personal dismay when Tuesday, October 30 at 11:34AM I got the following message from Squarespace:

I have some unfortunate news to share. Our primary data center, Peer1, in Lower Manhattan lost power yesterday at about 4:30PM local time. At that time, we smoothly made the transition to generator power and took comfort over the fact that we had enough fuel to last three to four days. (Peer1 stayed online during the last 3 major natural disasters in the area, including a blackout that lasted for days.)

At 8:30PM yesterday, we received reports that the lobby in the data center’s building was beginning to take on water. By 10:30PM, as is sadly the case in most of Lower Manhattan, Peer1’s basement had experienced serious flooding. At 5AM, we learned our data center’s fuel pumps and fuel tanks were completely flooded and unable to deliver any more fuel. At 8AM, they reported that the generators would be able to run for a maximum of four more hours.

Unfortunately, this means that Squarespace will be offline soon (our estimate being at 10:45 AM today).

I then did what any IT ops person would do…and notified my users of this outage:

Of course, I then did what any user would do, and emailed Squarespace support (like they had time for me.)

Can you guys toss up a graphic of some kind so people accessing my sites won’t get a dns error?

(Also, when you’re back there is nothing to do)?

An amazingly fast 26 minutes later, I had a response:

Great question! We will have a holding page up (hosted outside of Squarespace) that will provide messaging about the downtime. Any customers trying to access sites during that time will see that message. Once we are able to bring the system back up, there will be nothing required of you in order for your website to come back online. We expect sites to be available for another 45 minutes at least and please keep an eye for updates on Twitter (@Squarespace, @Squarespacehelp) as we will be providing updates as regularly as possible from there. 

Hope this helps!

Shaun H

Of course, being rather chatty, I responded with:

I will expect something very creative….

Like an overhead view of Sandy going down a toilet bowl. (Trying to bring a smile to your face during this tough time.)

Think the Squarespace version of the Twitter flying whale.

And again, had a quick response

Hey Gary,

That’s a great and hilarious suggestion, thank you :) We will definitely keep you updated on our Twitter accounts and Blog page (for as long as we can):

https://twitter.com/squarespace

and

http://blog.squarespace.com/

Hope this helps.

Paulina V.

What then followed was something I find speaks to the spirit of a team focused on service. 

They carried fuel to the generator on the roof.  17 floors.  All by hand.  Squarespace, their co-lo provider PEER1, another company Fog Creek (an online project management firm for collaborative software development) and some hired contractors carried the fuel to the 17^th floor where it could be pumped up to the generator on the roof (18^th floor).

Ok, let’s do some math.  According to PEER1’s Meredith Eaton, a company spokeswoman, the generator’s consumption rate was about 40 gallons/hour.  That’s eight 5 gallon pails an hour.  At 7.15 pounds/gallon diesel, that’s 286 pounds an hour up 17 floors.  And they did this for a couple days…so at 48 hours this is 13,728 pounds of fuel, or nearly 7 US tons of fuel.

The following pictures are used with permission of Squarespace:

 Thirsty generator, on the roof above 17 floorsBasement level, where the fuel is supposed to be stored

Diesel fuel on street waiting for a lift

Part of the bucket bridgade

Now, many would argue Squarespace would be better off with a second data center somewhere with automated failover.  That would carry an increased cost, something this author wouldn’t be willing to pay for.  Disaster recovery desires must be analyzed in light of the costs.  It’s almost laughable to consider a Recovery Time Objective or Recovery Point Objective for these blogs.  If they are down for days, frankly it wouldn’t matter.  These blogs are not time sensitive, with the closest financial impact being on the Mark Fidrych Foundation with donation ability (I encourage you to use!)

So due to the heroic efforts of the unsung IT Operations and associated people, PEER1 stayed up, and you are able to enjoy reading this post.

My hat goes off to these people who persevered, with determination and grit, to keep the site going.  In a word, amazing.  I find IT organizations do this often to keep the ship afloat, often without complaint.

Will we continue hosting on Squarespace?  You betcha.

What stories do you have of heroic IT efforts?

One midnight shot of a total bucket brigade

Another Open Letter to Co-Lo Data Center Operators

In this guest post, Matt Ferm shares more insights for co-lo operators:

On February 6^th Gary Kelley, posted an article titled “An Open Letter to Co-Lo Data Center Operators.”  In the article he offered some do’s and don’ts during the RFP process.  I would like to share some observations around what prospects are look for when evaluating a data center co-location provider.

As Gary mentioned, Harvard Partners performs a lot of co-lo RFPs, co-lo contract negotiations on behalf of clients, and migrations to co-location sites.  We coach and mentor clients from the point they think they might want to migrate their data center to watching them become fully operational in a co-location site.  During this process we see the reaction clients have to the sales pitch, value proposition, and site visit of many co-lo operators.

Here are some observations:

• You are not special – we know every co-lo operator is special and you are all better than your competition.  After two site visits (and two data center tours) our clients turn to us and say “it feels like we have done this before.”  As this is what you are selling, you must do the data center tour, but please recognize it can be mind-numbing to see one data center after another.  Keep the tour brief and focus on what will interest the prospect.
• Listen to the customer – the customer has selected to visit because they are trying to solve a specific problem.  Maybe it’s a construction project impacting their data center or frequent outages.  Maybe they want to instill more discipline in their data center operations.  In any case, it is important for you to understand who you are talking to and why they are seeking co-location.  Don’t compare the prospective customer to organizations they can’t relate to.  Using an example of a very large company when talking to a medium-sized prospect can become insulting as it makes the prospect feel much less important.
• Don’t oversell - most customers are coming from a server room or a Tier 1 data center.  Maybe they have a generator and maybe they have some extra cooling.  The likelihood is their “data center” is a converted room in a commercial office building.  You are about to show them something so far beyond their imagination they tend to get overwhelmed and ask “do I really need all this.”
• Details matter – customers come to you because they know you will pay attention to the details required to operate a state-of-the art data center.  Don’t create doubt with such incidents as dirty floors, too many construction workers around production equipment, ceiling tiles with water stains, visible rodent traps, and propane grills anywhere near your generators.  When the prospective customer comments “I wouldn’t let that happen in my data center” you know you’ve lost the deal.

Two propane grills (and a picnic table) adjacent to the diesel generators (and belly tanks containing fuel.)

• Make it real – whatever you sell make sure the prospective customer can translate it into dollars.  Every co-lo operator talks about managed services and cloud in one slide and then goes back to talking about generators, cross-connect rooms, and cooling towers.  If you are going to talk about extra services then make them real through a demonstration and consider bundling services into the co-lo contract.  Every one of our clients has told us co-location is the first step to the cloud.  We watched two CIOs “wake up” as a co-lo operator created a fully operational Windows Server instance in their cloud within 5 minutes.  The same co-lo operator offered free “cloud migration services” which got a response of “so what” from the CIOs.  When the offer changed to a free 4 CPU, 12 GB RAM, 100GB storage instance for a year, the CIOs immediately decided to go with that co-lo operator.  The CIOs were able to dollarize the offer and place a value on it.
• Become a partner – customers are looking for you to demonstrate how you will make them better.  Providing a resilient and robust data center is an important selling point, but so is your knowledge of process, managed services, technical architectures, cloud, and DR.  We watched as technical teams between a co-lo operator and a customer started to have a “mind meld” over a network architecture.  The CIO observed the interaction between the two teams and was convinced he had found a partner and made a decision to go with that co-lo operator.
• Be personable – Gary always reminds me “people buy from people.”  In the co-lo space it is always true.  You might think they are buying the generators, chillers, and multiple points of access into your facility, but what they are buying is you.  The CIO is looking for the person they will trust with their career.  Every CIO I meet when working on co-lo engagements tells me the co-lo decision is a make-or-break decision for their career.  Very simply, if they can’t provide services to their users, then they had better start updating their resume.
• Price is important – once you accept you are not special then you realize co-location services are a commodity (sorry).  Responding to an RFP with a low price gets the attention of CIOs.  Most of the time, before we get called in by a CIO, they have already visited one or two co-location sites.  They like what they see and are very excited, but realize they should do an RFP.  When one of those co-lo operators comes back with something resembling “list price,” the CIOs go visit other co-lo operators and quickly realize the services they require are commodity.
• Bundle – when we do an RFP we always ask for a certain number of remote hands hours to cover things like tape rotation.  You can differentiate yourself by offering these types of services as part of the base cost when the prospective customers visit you.  It is something they remember.

We believe the co-location market is changing.  Just two years ago we were mostly concerned about customer expansion over the course of a 5-year contract.  Today, we must think about contraction of space as customers think of going to managed services and the cloud.  Your relationship with the customer and helping them grow are the key selling points as our industry moves forward.

Indecision

In this guest post, Matt Ferm shares some observations around what drives good decisions.

High performance IT management, staff, and organizations know how to make decisions.  They are capable of quickly evaluating a situation, understanding decision criteria, assessing risk, and turning decision into action.  They produce results.

As infrastructure and operations consultants we see many IT organizations struggle to get projects, vendors, new employees, and purchases approved.  We consistently see this in the data center co-location selection process.  IT departments decide to migrate their in-house data centers to external co-location providers, have us conduct an RFP, find the right vendor, and then “shift to neutral” when it comes time for executive management to make a decision.  So, where do they go wrong?

IT organization tend to weigh the “what” (i.e., generators, cooling, etc.) higher than the “how.”  We tend to get distracted by “shiny objects” rather than understanding our users and decision makers.  We forget to ask ourselves the following questions before starting down the path of proposing something new:

• Who needs to make the decision?
• For each of those individuals, what is their criteria and process for making a decision?
• How do you take facts and feelings and make them actionable?
• What is a realistic timeframe for the decision?

When designing solutions, we have been taught to collect user requirements, create a design, build, test, and implement.  Decision making is no different.  Understanding the decision makers and their priorities is the equivalent of requirements gathering.  Determining the process for decision making is design, and turning facts and figures into actionable results is the build.  Effective decision making requires these steps before evaluating solutions.

Suggestions for improved decision making are:

• Early selection of decision makers and understanding the difference between your decision makers and “the decision makers”
• Receive approval from decision makers on a process and timeframe for decision making
• Insure the desired business outcome is relevant to the decision makers (a data center co-location decision is the same as an office lease to the CFO)
• Select one decision maker and pre-sell your proposal as a test to see how others will react
• Never use numbers (spreadsheets) to sell an idea.  Numbers should support a solid business case
• No one likes being pushed to make a decision.  Set realistic timeframes and expect iteration.
• Prepare for and do not over-react to the “devil’s advocate”
• Identify your harshest critic and target your “sale” to them.  Convince them and the rest will follow.

When a decision does not go my way (and I know my proposed solution is correct), I see it as my failure for not being able to convince someone of my beliefs.  In the end, your ability to effectively communicate opportunities and costs is what will lead to quicker decisions.  Stay objective and put yourself in the place of the person making the decision, and you will find yourself on the path to your desired outcome.

An Open Letter to Co-Lo Data Center Operators

Regular readers know I am a fan of using co-location providers for small and mid-size businesses.  Larger businesses have the mass and scale to run their own data centers.

We are seeing a dramatic uptick in the number of organizations seeking co-location.  Some are moving because of external factors (the CIO with an elevator going through the data center opted to go co-lo, as did a CIO where flooding damaged the data center building.)  Others are experiencing growth, and finding the costs to expand in place prohibitive.

Whatever the reason, these organizations often look to us to provide advice and counsel.

As a part of the co-location decision, we believe in an RFP process.

Analyzing hundreds of responses leads us to make a few observations:

• Always be honest – we had a respondent claim to be a Tier 4 (think NYSE and the government) data center operator.  Tier 4 has redundant everything….real belt and suspenders.  For this respondent, it would be a neat trick as they only had one generator.

Were they intentionally misleading, or did they not understand? 

• Check your math – As independent consultants, we analyze the financial aspects of proposals using proprietary models we’ve built and refined.  As we load vendor’s submittals to our spreadsheet, we often find errors in submissions (most often, the words and the numbers do not align.)
  
  
• Respect the process – it seems every one of our proposal efforts exposes a vendor who is unwilling to follow the process.  This normally takes the form of a vendor reaching around the consultant (us) and trying to influence directly or indirectly.  Perhaps this is the equivalent of a Hail Mary football pass, a final act of desperation…since in all cases the vendor going around the consultant did not prevail in the end on the strength of their response.  We know vendors will do this, and so we prepare our clients for the defense.

We work very hard for the client and the respondents to play fairly.

We recommend our clients visit co-location providers and “kick the tires” before making a final decision.

This is where gaps in the expectation versus delivery are often exposed, yielding more observations.  I like getting to the co-lo site an hour before the formal tour, and take a walk around the building.  I take my trusty BlackBerry with me, and take pictures of things I see.  In fairness, I am not including pictures in this blog post intentionally as I don’t want any vendor feeling bad, or any vendor using this to market against another vendor.  That said, what I find is vendor seem so driven to please they overstate the facts:

• “You wouldn’t be in our parking lot for more than 5 minutes without someone challenging you.”  Well, I sure took a lot of pictures in the 15 minutes I was out there.
• “The gas main at the front door has been deemed ‘Not A Risk.’”  I don’t know…it seems a gas main should be hidden/protected from automobiles.
• “We check our logs closely.”  OK, good.  (And Bill Gates just signed in.)
• “We would never let a constable take anything, even if they had appropriate court papers”  The constable sitting in the room did not agree.
• “You wouldn’t get close enough to see the equipment.  You would be stopped in 3 seconds.”  Well, I have pictures of the serial number right off the device, with my little non-telephoto equipped BlackBerry.
•  “That door to the UPS Switchgear area is never opened.”  We have a picture of it propped open.  There was someone standing there when it was open.  Really?

To be clear, other than forging Bill Gate’s name we don’t do anything illegal.  We’re not scaling fences, or using ladders.  Just a nice stroll around the building.

One client recently insisted in participating by taking a water bottle with them on tours.  While only one vendor didn’t challenge the bottle, that one vendor lost a large amount of credibility.

I can’t stress enough the importance of keeping a facility neat, tidy and tour ready.  Seeing a cage in a co-location facility used as a janitor’s closet leaves me scratching my head.  Having all the covers off HVAC devices with nobody working on them makes one wonder about the overall maintenance plan.

If a client asks to see the mechanicals, show them.  We recently had one vendor that hesitated, and this left a sour taste in the client’s mouth.  The vendor did eventually show the space…

Power is an important element of cost.  Vendors putting all their profit in the power creates a disparity with others.

One or two vendors like to trash their competitors.  We’re not fans of that approach.  Speak to your own merits.

One last thing we tell our clients is to check references, and to Google the vendor.  Ask if there have been business impacting outages, and how they are to work with.  If a vendor has had a data center floor impacting issue, hopefully they’ve already mentioned it, and how they incident management process quickly addressed, and steps taken to address.  This is electro mechanical gear, and things CAN happen.  Rarely!

Evaluating Co-Location Data Centers

Certainly by now you know I’m a fan of co-location.  Considering co-location is a must for any organization with a data center need under 20,000 square feet.

Having determined requirements and engaged co-location providers with an RFP, it’s now time to narrow the focus and make a decision.

Using an evaluation matrix is a useful way to narrow the herd.   In the first column, indicate your evaluation criteria (location, price, total capability, etc).  Each criterion is then assigned a weight, with 1 being the lowest and 10 being the highest.  Each vendor is then evaluated on a 1-10 scale, with the resulting score = weight x eval.   Those scores are then added for a final…allowing comparison.

 

While I am a fan of an evaluation matrix, having performed this exercise dozens of times there are some issues with this approach worth discussing.

If a vendor simply doesn’t meet the requirements, they get a “low” score, yet not meeting requirements should be sufficient to disqualify.  If ALL vendors miss on requirements, the chances are good either the requirements or RFP are “too tight.”  It may also be indicative of a need to do an in-house data center….and in practice we rarely see this.

What is more likely is the vendors will end up being in groups.  We use these groups for our analysis and next steps.  The “top” group will be the ones to continue conversations with, and the bottom group is most likely a group worth passing on.

Here’s the dirty little secret.  The evaluation matrix gives an objective appearance to a subjective process.  You still need to go out and visit the (short) list of companies before making a final decision.  We use the evaluation matrix as a tool to narrow the list, not the ultimate decision tool.

Once you’re met with vendors at their location and kicked the tires, a final evaluation matrix can be assembled.

When you visit the co-location providers, take time in your evaluation.  The ANSI/TIA standard 942 can be used as a guideline, or professional services can assist.

 Walk around the building and get a general sense of the neighborhood and plant housekeeping.  You should not be able to get close to the operating equipment at street level.

When you tour the data center, is security reasonable for your company? 

Spend time understanding the Mechanical, Electrical and Plumbing (MEP) parts of the data center…this is the “guts” and arguable most important part of the facility.  Is the housekeeping of the MEP pristine?   I’m a big believer in taking my old car for service in the ugliest garage in town…and the MEP area shouldn’t look like that!   If you see buckets under leaking valves, you can quickly realize maintenance isn’t what it should be. 

Ask the hard questions….ask to see evidence of equipment maintenance (we visited a co-location provider touting regular infrared electrical panel scanning, and the last time it was performed was three years previously.)  Ask for a history of outages (you may be surprised by what you discover.) 

Take time to understand the communications carriers already in the building.  Do not assume your carriers are already there.  Co-location providers have “meet me” areas, where carriers and customers are interconnected.  Cable management in this space is imperative. 

Ask to speak to references…and do so.  Find out if the provider is easy to work with, or a fan of up- charging for everything.  Let’s face it, your equipment will break, and you may need to get a vendor to ship a part; you shouldn’t have to pay for “processing a delivery!”

Spend time on Google, and find out if there are any articles about outages/issues in the facility.

I’m a big believer in the end people buy from people…so don’t hesitate to think about whether you can simply work with the provider and their people.  People will change from time to time, and the “move in” period is a very important time.

Look at the contracts.  All will have protections for the provider for outages; it’s how they deal with them you need to consider.

Only when you have completed your due diligence can you make your final decision.

How to Select a Co-Location Provider using an RFP

In a recent post, the case was made for companies with modest data center needs to explore co-location.

We made our case and the executive team agreed.  Having an agreed upon direction, it was now time to do something.  And the client was stumped.

“Let’s go visit them,” the enthusiastic client espoused. 

We would recommend visiting a couple to ground yourself in some of the various dimensions…and then would quickly go back to a conference room to understand requirements.

“Our systems are important.  We need to put everything in a Tier 4 data center.”

The ANSI/TIA-942 standard defines the concept of Tiering, as a way to distinguish design considerations.  The standard is quite lengthy, and is summarized below.  You’ll see “N” mentioned.  “N” stands for NEED….and is always debated since everything else drives from it.

	Tier 1	Tier 2	Tier 3	Tier 4
Delivery Paths	One	One	One Active One Alternate	Two Active
Capacity Components of Support	N	N+1	N+1	N after any failure
Points of Failure	Many + Human Error	Many + Human Error	Some + Human Error	Fire, EPO & Human Error
Yearly Downtime	28.8 Hours	22.0 Hours	1.6 Hours	0.8 Hours
Site Availability	99.67%	99.75%	99.98%	99.99%

While there are some commercially available Tier 4 co-location facilities, unless there is a specific business requirement, we find most co-location facilities at a Tier 3 level provide the kind of availability one would expect for a paid service.  The yearly projected downtime between a Tier 2 and a Tier 3 is substantial (don’t assume the “yearly downtime” happens once a year.  For example, what if it worked out the 22 hours (Tier 2) was spread over a year in weekly increments?  While the facility would be hitting its uptime goal, your systems outage time (quiescing applications, database, and systems, rebooting, then bringing up systems, databases and applications) could add to hours each week!)

With a good understanding of the capabilities available and your business needs, a Request for Proposal (RFP) can be put together.

Putting together an RFP can be a fun task, or maddening.  The key to the RFP process is using the development period to driving alignment of the company parties. 

At the most basic level, the RFP should cover:

• Company Background (don’t assume people know your company)
• Stated Requirements (high level)
• Growth considerations (this is the hard part….and often drives initial buildout costs)
• Key Questions (on how delivery is provided)
• Response format (similar response formats make comparisons easier)
• Any specific legal topics (such as a need for background checks)

There is a fine balance in writing RFPs.  Companies need to be specific enough deliverable solutions can be proposed, yet broad enough to allow creativity.

For example, one area driving costs in RFPs is around power/cooling.  The company needs to identify what the heat load is of the equipment envisioned to be placed, in kilowatts (kW). 

If the company is running a dense environment, it is tempting to express this in watts per square foot (W/SF)

When W/SF are used, some providers may be automatically “disqualified.”   How?  A data center designed for 50W/SF can indeed support 100W/SF….it just takes twice the space and appropriate chilled air distribution! 

So smart companies analyze their needs, and let the co-location providers respond.

Philosophically, we prefer shorter RFPs to longer ones.  Providers need to have time to put together responses.  If the RFP is “too big, too heavy,” some providers may not respond at all or will respond generically with their capabilities, while not answering the underlying questions in the RFP.

Once the RFPs hit the street, you need to put a cone of silence on vendor conversation at ALL levels.

Next week, we’ll talk about how to analyze the responses.

Have a favorite RFP story?  Share it with the community.  We’d suggest masking company names…