Want email updates from me?
Want more unvarnished truth?
Looking for something? Look here!
What I'm saying now
What you're saying...
I think tag clouds are pretty, and not to be taken overly seriously
111 Chop House 75 on Liberty Wharf 9/11 A Broth of a Boy ABCs Abiouness accountability activities alcohol Allora Ristorante Analysis Angry Hams ANSI/TIA 942 Anthony's Pier 4 Apple Application Armsby Abbey Arsenal Arturo's Ristorante Ashland AT&T Audio Automation baby Baby Monitor babysitting Back To School Bad News Bangkok Thai Bar Bay State Common baystateparent BBQ BCP BeeZers Before I die I want to... behavior Big Bang Bike Bill of Rights Bistro Black Box BlackBerry Boston Boston Marathon boundaries Boyston BPO brand Breakfast Bridge Bring Your Own Technology Budget Burlington Burn Burrito buyer BYOD Cabling Cambridge Camp Campaign career Casey's Diner Castle casual cCabling Cell Phone Central Square Change Management Cheers Chef Sun ChengDu Chet's Diner Children Chinese Christmas Christmas Families Holiday CIO Cloud coddle collage College College Acceptance co-lo Co-Location Co-Location Tier Power Cooling Comfort Food Control Country Country Kettle Crisis customer dad Dad Phrases damage daredevil Data Center Data Center Design Davios Day Care Dead Death declaration Del Frisco's Design Desktop Video dinner Disaster Recovery Divorce Do Epic Shit dodgeball Downtown Crossing DR driving Droid Easter Economic Kids Edaville Education Elbow Night Elevator Employee Engagement Erin Etiquette Evaluation events Exchange Expiration Dates Facebook Failing family Fatherhood Favorite things Flash Flemings Fogo de Chão Food Hits and Misses Format Foundry on Elm Foxborough Frameworks fraternity Fraud French Fried Clams friends fun Fusion Generations germs Girl Scouts girls Global Go/No Go GPS Grafton Grandchild Grandpa Harry's hazing Healthcare Healthy Choices while Dining Out Help Desk Hisa Japanese Cuisine Historic holiday Home hope Horizons hose Hot Dog Hurricane IIT Assessment incident Indecision Indian Infrastructure Inn Innovation Internet Inventory Management iPhone IT IT Assessment IT Satisfaction Italian Jack Daniels Jakes Restaurant Janet Japanese Jazz Joey's Bar and Grill JP's Khatta Mitha kickball kids Laid off Lakes Region Lala Java Leadership Learning legacy Legal Harborside Les Zygomates L'Espalier Liberty Wharf lights out Linguine's loss Love Lucky's Cafe M&M Macys Thanksgiving Day Parade mai tai Managed Application Services Managed Services managers Mandarin Manners Mark Fidrych marriage Mary Chung mass save Maxwell-Silverman Mediterranean meetings Memorial Day memory Mendon Mergers Mexican MiFi Migration Ming III miss MIT MIT CIO Symposium Mobility Moes Hot Dog Truck MOM money Mother Moving on Name neanderthal neighborhood Network New York Marathon newborn Northborough Not Your Average Joe's Nuovo Nursing On-Call Operations Operators Oregon Club Organization Pancakes Pandemic Parental Control Parenting Patch Peeves People Perserverance UMASS growth Play Plug and Run Predictable Pride Problem Process Production program Project Management propane PTA. PTO PUE QR Quick Response Rant Real Estate Realtor Recognition Red Rock Resiliency Respect restaurant Restaurant Guy RFP ribs Ritual Root Cause Analysis Sam Adams Sandy Sapporo savings School Sea Dog Brewing Company Sea Dog Steak and Ale Seafood Seaport Security Sel de la Terra Service Service Desk Service Indicator Light sharing SHIRO Shit Pump Shriners SHTF Simplification Skunk Works Skype Sleep sleepovers Sloan Smith & Wollensky soccer Son SOP sorority spanking Squarespace staffing Starbucks Status Reporting Steak Steve Jobs Storage Strategy stress Summer Sushi swimming Tacos Acalpulco teacher Technology Teen Telephony Temperature Strip Tenka terrorist Testing Texas BBQ Company Text Thai Thanksgiving in IT The Mooring Thomas Thought Leader Three Gorges III TIA 942 Timesheets Toby Keith Toddlers traditions Transition treehouse turnover TV Twitter unspoken moments Valentine's Day Value Vendor Venezuelan Verizon Vermont Video Vietnamese voice VoIP Watertown Wedding Westborough Korean Restaurant Westborough MA. StormCam WiFI Wi-Fi Wilbraham Wine Worcester work work life balance working Yama Zakura Zem Han Zitis

Entries in Migration (2)

Monday
May062013

The Importance of Staff & Shifts

In the course of our business, we see many data center/applications migrations and/or high-severity issues.  One observation we always share with our clients is to plan for staff rotation.  As you might expect, some listen and others do not. Here’s why it’s important.

Migrations often happen overnight…when the business sleeps or operates at a lower activity level.  Organizations without satisfactory disaster recovery plans often incur an outage to do a migration.  People are resilient for so many hours, and then they crash. 

What often happens in migrations is everyone wants to be at the starting line, and the adrenaline keeps them engaged.  If shifts are not “forced,” then there is often nobody left with “gas in their tank” to troubleshoot issues.  People simply have to disengage to be fresh.

We saw this at a large customer where the team had persevered, declared success, and then dragged themselves home.  There was an issue, and the on-call was unwilling to make changes as he didn’t understand the changes that had taken place (a change management issue.)  NOBODY involved was responding to calls.  As it turned out, the group’s manager lived in my town, and I got to knock on his door at 10:00AM on a Sunday morning.  His wife wasn’t happy (he had been up all night) and did indeed get him up.  While he resolved the issue, a few months later he resigned and went to work at a different company. 

In this case, the team was not structured to focus on a multiple day issue….and response was poor.

In another case, a new virus definitions in client’s antivirus system determined the operating system was bad, quarantining the operating system.  The client had a policy to delete quarantined files, so with the speed of automation thousands of operating systems were deleted.

The senior manager quickly determined this would require a sustained 24/7 response, and teams were “nominated” to cover 12 hour shifts.  We were asked to help on a sustained basis, providing process oversight and helping with crisply doing turnovers.

To the credit of the senior manager, this approach allowed a sustained response as systems we recovered from (gasp!) tape.

Large IT shops often run with multiple shifts and a technical response is more organic.  Smaller shops tend to have an operational capability 24x7, and may lack the detailed technical response.

When planning or reacting to major events, think in terms of how to rotate your staff for a sustained time.

Monday
Jul042011

Data Center Migrations

As companies emerge from the great recession, data center spending appears to again by in vogue.

While we believe colos and clouds deserve serious attention, many companies are expanding existing data centers to address the needs of business and/or systems growth.

Ok not THAT kind of migration

The first, and arguably most important, step in any migration effort is understanding the inventory. It’s our experience few companies have this airtight.

The inventory must include hardware (model, serial number) and also applications, databases, circuits and interdependencies. Any special power (including outlets) and cooling needs (a blade enclosure brings a significant heat source.) Network and SAN connections must be understood.

The company then decides how to perform the migration (most companies decide this first, and we believe in understanding the requirements first). Is it a “lift and shift,” a P2V (physical to virtualize) effort, or a fork lift upgrade? How the data will be migrated is a key consideration, with attention paid to how quickly data can be moved.

Most companies allow development machines to be “moved” anytime, although consideration must be given to the development machine usage. For example, a US based company moving a development environment during the US day is preferable when using a India development team.

The network team is a key player in any migration. How the network is extended between locations will have a direct impact on how challenging the migration will be….especially if IP addresses must change. Any administrator will tell you changing IP addresses is an easy task; however any “bad” application coding techniques will quickly be exposed.

Arguably the easiest migration is establishing a mirror environment, migrating data and testing applications in a “cocoon” segregated from the current production environment. This type of migration reduces downtime, and if clustering approaches are used downtime is virtually eliminated. Migrations are also an opportunity to use disaster recovery environments (provided they are well tested in advance.) The target environment can be an upgraded processing environment, and vendors often facilitate with an asset swap minimizing licensing and tax ramifications.

Perhaps the most challenging migrations are lift and shifts. In these migrations, a (backed up) environment is brought down, moved, and then restored. The issue is “falling back” is a practical challenge. Once the machines are moved, it’s very rare they are ever moved back…instead, they are “fixed in place.”

It’s important as much testing and verification as possible is performed before moving systems. All electrical must be confirmed, cables/fiber tested and labeled, door widths and heights reviewed, physical security access confirmed, loading dock heights confirmed, etc. When you find that one loading dock door that’s “always open” locked…how do you get it unlocked at 2AM? And when you are moving distances, it is prudent to check weather and road closures whenever possible BEFORE starting the migration. Some equipment is so large, professional movers are needed.



In a physical move, the application is brought down, backups performed, database brought down, followed by the server being brought down. Once at the target location, the server is brought up by the system administrators, database is confirmed by the DBA team, and then the application is prepared for user verification BEFORE productive use. At best, this is a multi-hour event. In a virtual migration, data is first migrated, and so the live environment is established and tested. The network team can then repoint DNS so the new environment. The application should be quiesced in the original environment so nobody can continue entering transactions.

Migrations can take time and often take place in the wee hours. It’s our experience, especially when physically moving older equipment; some equipment will not immediately come back (~1%). This often requires an extended recovery time, and necessitates “shifts” of support people. Technicians need to know this in advance and plan for it; to many, it is a badge of honor to work 72 hours straight.

In a short piece like this all options for migrations can’t be covered. Migrations are projects like any other, and require good planning and execution.