The Importance of Staff & Shifts

In the course of our business, we see many data center/applications migrations and/or high-severity issues.  One observation we always share with our clients is to plan for staff rotation.  As you might expect, some listen and others do not. Here’s why it’s important.

Migrations often happen overnight…when the business sleeps or operates at a lower activity level.  Organizations without satisfactory disaster recovery plans often incur an outage to do a migration.  People are resilient for so many hours, and then they crash. 

What often happens in migrations is everyone wants to be at the starting line, and the adrenaline keeps them engaged.  If shifts are not “forced,” then there is often nobody left with “gas in their tank” to troubleshoot issues.  People simply have to disengage to be fresh.

We saw this at a large customer where the team had persevered, declared success, and then dragged themselves home.  There was an issue, and the on-call was unwilling to make changes as he didn’t understand the changes that had taken place (a change management issue.)  NOBODY involved was responding to calls.  As it turned out, the group’s manager lived in my town, and I got to knock on his door at 10:00AM on a Sunday morning.  His wife wasn’t happy (he had been up all night) and did indeed get him up.  While he resolved the issue, a few months later he resigned and went to work at a different company. 

In this case, the team was not structured to focus on a multiple day issue….and response was poor.

In another case, a new virus definitions in client’s antivirus system determined the operating system was bad, quarantining the operating system.  The client had a policy to delete quarantined files, so with the speed of automation thousands of operating systems were deleted.

The senior manager quickly determined this would require a sustained 24/7 response, and teams were “nominated” to cover 12 hour shifts.  We were asked to help on a sustained basis, providing process oversight and helping with crisply doing turnovers.

To the credit of the senior manager, this approach allowed a sustained response as systems we recovered from (gasp!) tape.

Large IT shops often run with multiple shifts and a technical response is more organic.  Smaller shops tend to have an operational capability 24x7, and may lack the detailed technical response.

When planning or reacting to major events, think in terms of how to rotate your staff for a sustained time.

Data Center Migrations

As companies emerge from the great recession, data center spending appears to again by in vogue.

While we believe colos and clouds deserve serious attention, many companies are expanding existing data centers to address the needs of business and/or systems growth.

Ok not THAT kind of migration

The first, and arguably most important, step in any migration effort is understanding the inventory. It’s our experience few companies have this airtight.

The inventory must include hardware (model, serial number) and also applications, databases, circuits and interdependencies. Any special power (including outlets) and cooling needs (a blade enclosure brings a significant heat source.) Network and SAN connections must be understood.

The company then decides how to perform the migration (most companies decide this first, and we believe in understanding the requirements first). Is it a “lift and shift,” a P2V (physical to virtualize) effort, or a fork lift upgrade? How the data will be migrated is a key consideration, with attention paid to how quickly data can be moved.

Most companies allow development machines to be “moved” anytime, although consideration must be given to the development machine usage. For example, a US based company moving a development environment during the US day is preferable when using a India development team.

The network team is a key player in any migration. How the network is extended between locations will have a direct impact on how challenging the migration will be….especially if IP addresses must change. Any administrator will tell you changing IP addresses is an easy task; however any “bad” application coding techniques will quickly be exposed.

Arguably the easiest migration is establishing a mirror environment, migrating data and testing applications in a “cocoon” segregated from the current production environment. This type of migration reduces downtime, and if clustering approaches are used downtime is virtually eliminated. Migrations are also an opportunity to use disaster recovery environments (provided they are well tested in advance.) The target environment can be an upgraded processing environment, and vendors often facilitate with an asset swap minimizing licensing and tax ramifications.

Perhaps the most challenging migrations are lift and shifts. In these migrations, a (backed up) environment is brought down, moved, and then restored. The issue is “falling back” is a practical challenge. Once the machines are moved, it’s very rare they are ever moved back…instead, they are “fixed in place.”

It’s important as much testing and verification as possible is performed before moving systems. All electrical must be confirmed, cables/fiber tested and labeled, door widths and heights reviewed, physical security access confirmed, loading dock heights confirmed, etc. When you find that one loading dock door that’s “always open” locked…how do you get it unlocked at 2AM? And when you are moving distances, it is prudent to check weather and road closures whenever possible BEFORE starting the migration. Some equipment is so large, professional movers are needed.

In a physical move, the application is brought down, backups performed, database brought down, followed by the server being brought down. Once at the target location, the server is brought up by the system administrators, database is confirmed by the DBA team, and then the application is prepared for user verification BEFORE productive use. At best, this is a multi-hour event. In a virtual migration, data is first migrated, and so the live environment is established and tested. The network team can then repoint DNS so the new environment. The application should be quiesced in the original environment so nobody can continue entering transactions.

Migrations can take time and often take place in the wee hours. It’s our experience, especially when physically moving older equipment; some equipment will not immediately come back (~1%). This often requires an extended recovery time, and necessitates “shifts” of support people. Technicians need to know this in advance and plan for it; to many, it is a badge of honor to work 72 hours straight.

In a short piece like this all options for migrations can’t be covered. Migrations are projects like any other, and require good planning and execution.