Looking for something? Look here!
Want more unvarnished truth?
What you're saying...
What I'm saying now
I think tag clouds are pretty, and not to be taken overly seriously
##MoveWithGary #Home Inspection #MoveWithGary 111 Chop House 75 on Liberty Wharf 9/11 A Broth of a Boy ABCs Abiouness accountability activities alcohol Allora Ristorante Analysis Angry Hams ANSI/TIA 942 Anthony's Pier 4 Apple Application Armsby Abbey Arsenal Arturo's Ristorante Ashland AT&T Audio Automation baby Baby Monitor babysitting Back To School Bad News Bangkok Thai banks lending movewithgary Bar Bay State Common baystateparent BBQ BCP Bees BeeZers Before I die I want to... behavior Big Bang Bike Bill of Rights Bistro Black Box BlackBerry Boston Boston Marathon boundaries Boyston BPO brand Breakfast Bridge Bring Your Own Technology Budget Burlington Burn Burrito buyer BYOD Cabling Cambridge Camp Campaign career Casey's Diner Castle casual cCabling Cell Phone Central Square Change Management Cheers Chef Sun ChengDu Chet's Diner Children Chinese Christmas Christmas Families Holiday CIO Cloud coddle collage College College Acceptance co-lo Co-Location Co-Location Tier Power Cooling Comfort Food Control Country Country Kettle Crisis customer dad Dad Phrases damage daredevil Data Center Data Center Design Davios Day Care Dead Death declaration Del Frisco's Design Desktop Video dinner Disaster Recovery Divorce Do Epic Shit dodgeball downsizing Downtown Crossing DR driving Droid Easter Economic Kids Edaville Education Elbow Night Elevator Employee Engagement Erin Estate Planning Etiquette Evaluation events Exchange Expiration Dates Facebook Failing family Family Law Fatherhood Favorite things first time buyer Flash Flemings Fogo de Chão Food Hits and Misses Format Foundry on Elm Foxborough Frameworks fraternity Fraud French Fried Clams friends fun Fusion Generations germs Girl Scouts girls Global Go/No Go GPS Grafton Grandchild Grandpa Harry's hazing Healthcare Healthy Choices while Dining Out Help Desk Hisa Japanese Cuisine Historic holiday Home Home Inspection hope Horizons hose Hot Dog Hurricane IIT Assessment incident Indecision Indian Infrastructure Inn Innovation Insurance Internet Inventory Management iPhone IT IT Assessment IT Satisfaction Italian Jack Daniels Jakes Restaurant Janet Japanese Jazz Joey's Bar and Grill JP's Khatta Mitha kickball kids Laid off Lakes Region Lala Java Leadership Learning legacy Legal Legal Harborside Les Zygomates L'Espalier Liberty Wharf life transition lights out Linguine's loss Love Lucky's Cafe luxury luxury home M&M Macys Thanksgiving Day Parade mai tai Managed Application Services Managed Services managers Mandarin Manners Mark Fidrych marlborough marriage Mary Chung mass save Maxwell-Silverman Mediterranean meetings Memorial Day memory Mendon Mergers Mexican MiFi Migration Ming III miss MIT MIT CIO Symposium mmortgage Mobility Moes Hot Dog Truck MOM money mortgage Mother MoveWithGary Moving on Name nature neanderthal neighborhood Network new listing New York Marathon newborn Northborough Not Your Average Joe's Nuovo Nursing On-Call Operations Operators Oregon Club Organization Pancakes Pandemic Parental Control Parenting Patch Peeves People Perserverance UMASS growth Photography Play Plug and Run Predictable Pride Problem Process Production program Project Management propane PTA. PTO PUE QR Quick Response Rant re/max Real Estate Realtor Recognition Red Rock Resiliency Respect restaurant Restaurant Guy RFP ribs Ritual Root Cause Analysis rReal Estate Sam Adams Sandy Sapporo savings School Sea Dog Brewing Company Sea Dog Steak and Ale Seafood Seaport Security Sel de la Terra Service Service Desk Service Indicator Light sharing ShearTransformation SHIRO Shit Pump Shriners SHTF Simplification Skunk Works Skype Sleep sleepovers Sloan Smith & Wollensky soccer Son SOP sorority spanking Squarespace staffing staging Starbucks Status Reporting Steak Steve Jobs Storage Strategy stress Summer Sushi swimming Tacos Acalpulco teacher Technology Teen Telephony Temperature Strip Tenka terrorist Testing Texas BBQ Company Text Thai Thanksgiving in IT The Mooring Thomas Thought Leader Three Gorges III TIA 942 Timesheets Toby Keith Toddlers traditions Transition treehouse turnover TV Twitter unspoken moments Valentine's Day Value Vendor Venezuelan Verizon Vermont Video Vietnamese voice VoIP Watertown Wedding Westborough Korean Restaurant Westborough MA. StormCam WiFI Wi-Fi Wilbraham Wine Worcester work work life balance working Yama Zakura Zem Han Zitis

Entries in Production (2)


When to Update Production and DR

Some companies run a “production only” environment.  Think a restaurant, where they buy packaged software and can use paper as a backup system.  The chances are excellent they buy packaged software, and are looking to the software provider to have proper systems management.

Other companies can take environments to another extreme…having multiple development, certification, integration, performance, staging production and disaster recovery environments, steadfastly promoting code through each environment.  Many organizations take “release management,” to levels comparable to large software firms.

Of course, the age old debate of few formal releases vs quick regular releases is always in play.

This post tackles a simple question.  Assuming you have both a production and a disaster recovery environment, what is the upgrade order?


Obviously the circumstances in your environment will drive what you do.  One might argue in a truly active:active environment, there is no such concept as disaster recovery.  For purposes of this post, production and DR are separate, failover is possible, and failover is tested regularly.

Some people argue the natural upgrade approach is to upgrade disaster recovery first, then complete an upgrade process by touching production.  In this approach, every stage of the promotion process is views as tests preserving production for the “final” change.

We posit Disaster Recovery should be upgraded after ensuring Production is stable.

It’s not that we don’t think Production is important.  To the contrary, we revere the Production environment.

By upgrading DR after Production, you assure the business can failover if the upgrade proves untenable in production.  There is always a known working copy available.

IT professionals following this approach have to determine when Production is stable.  Is it an hour?  A day?  A cycle?

We suggest it is after a day’s stable processing.  A day is arbitrary; as a practical matter once changes are in place there is a point of no return where any fixes will be made in the new environment and not after failing back.

IT professionals must remember a promotion cycle is not complete until DR is upgraded.  When organizations neglect to upgrade disaster recovery, they lose their failover ability.

Is this a once size fits all recommendation?  No, you need to look at your environment and the changes underway.  Database Schema changes, and core functionality changes may preclude a phased approach.  Since we fundamentally don’t subscribe to big bang, we suggest always trying to maintain a failover ability.

How does your organization deal with upgrades/migrations to minimize risk?


The Insanity Must Stop

“The insanity must stop,” the newly minted IT director shouted in shear frustration.
After numerous outages caused by people making mistakes combined with some equipment malfunctions, the IT team was just beaten down. .

Too many long days, combined with long nighttime problem resolution conference calls, prompted a vocalization capturing what everyone was thinking. Saying it out loud informally made it OK to discuss the situation and even chuckle about it.

“Being lucky” often means covering the contingencies so when things go awry the organizational and computerized systems can recover gracefully. When you find yourself in a “bad patch,” don’t invocate “Murphy’s Law” as the culprit. The culprit is…you.

You are positioned and expected to lay out the processes and procedures to enable stability. A “Production Stabilization Program” is often indicated when the organization has grown very quickly and/or implemented more change than the organization/systems can tolerate.

Understanding what is going on in a “bad patch” is critical. Analyze logs and incident reports identifying common themes and root cause. Talk to your staff about what they are feeling and seeing, and what solutions they may offer.

It is often useful to categorize the situation using a cause and effect diagram (Fishbone / Ishikawa ) or borrowing from a SWOT analysis (SWOT is as an acronym for Strengths, Weaknesses, Opportunities, and Threats, categorized by internal vs. external factors.) Such a categorization allows you to visually identify the opportunity areas for the organization. On this, the tool selection is far secondary to the capturing and categorization.

With the knowledge of the impacting factors, plans can be put in place to address. Some of the fixes may be quick (i.e.: we need to reprovision some storage to address the failures) and others may become longer term, strategic initiatives (i.e.: we need to implement a disaster recovery strategy we can use “daily” mitigating outages). In this case, having a plan and implementing becomes a very sensitive issue, and one teams can rally behind.

We often hear it’s hard to be responding on a daily basis to issues and working what’s essentially a separate project to identify impacts and develop plans to address. Managers need to be sensitive when the long hours are taking a toll…when staff becomes “snappy” or “grumpy”, it’s not the best time to add work. Often a fresh set of eyes and external perspective is invaluable.

Anyone being brought to bear on Production Stabilization Programs needs to have the professional maturity and sensitivity to be perceived as not out to “shoot the messenger” or solve all the problems of the world. They are working through a separate process, and their role is about process. If the staff misunderstands this, morale will suffer substantially!

The root cause identification and articulation (at a high, process level) and subsequent planning effort can be best put together by a team external to the issues, clearly with input from the affected team, with a review and vetting by the team for management consumption.

So when you are tempted to enlist Murphy’s Law as the cause, remember what Abe Lincoln said, “when you’re being run out of town on a rail, get in front of the crowd and make it look like you’re leading the parade.” Take the bull by the horns and lead the team to stability!