When to Update Production and DR

Some companies run a “production only” environment.  Think a restaurant, where they buy packaged software and can use paper as a backup system.  The chances are excellent they buy packaged software, and are looking to the software provider to have proper systems management.

Other companies can take environments to another extreme…having multiple development, certification, integration, performance, staging production and disaster recovery environments, steadfastly promoting code through each environment.  Many organizations take “release management,” to levels comparable to large software firms.

Of course, the age old debate of few formal releases vs quick regular releases is always in play.

This post tackles a simple question.  Assuming you have both a production and a disaster recovery environment, what is the upgrade order?

 

Obviously the circumstances in your environment will drive what you do.  One might argue in a truly active:active environment, there is no such concept as disaster recovery.  For purposes of this post, production and DR are separate, failover is possible, and failover is tested regularly.

Some people argue the natural upgrade approach is to upgrade disaster recovery first, then complete an upgrade process by touching production.  In this approach, every stage of the promotion process is views as tests preserving production for the “final” change.

We posit Disaster Recovery should be upgraded after ensuring Production is stable.

It’s not that we don’t think Production is important.  To the contrary, we revere the Production environment.

By upgrading DR after Production, you assure the business can failover if the upgrade proves untenable in production.  There is always a known working copy available.

IT professionals following this approach have to determine when Production is stable.  Is it an hour?  A day?  A cycle?

We suggest it is after a day’s stable processing.  A day is arbitrary; as a practical matter once changes are in place there is a point of no return where any fixes will be made in the new environment and not after failing back.

IT professionals must remember a promotion cycle is not complete until DR is upgraded.  When organizations neglect to upgrade disaster recovery, they lose their failover ability.

Is this a once size fits all recommendation?  No, you need to look at your environment and the changes underway.  Database Schema changes, and core functionality changes may preclude a phased approach.  Since we fundamentally don’t subscribe to big bang, we suggest always trying to maintain a failover ability.

How does your organization deal with upgrades/migrations to minimize risk?