The moral of Hostway's data center migration story

I really feel for my friends at Hostway. They're good guys who love what they do. John Martis puts a huge amount of effort into getting to know his customers, hence Hostway's ability to turn $20 shared hosting accounts into 100-server relationships. And John Lee is super generous about sharing results from his ongoing marketing experiments.

As you may have heard, Hostway - and a number of its customers - had quite a rough time last week. During a Miami-to-Tampa migration, an unexpectedly large number of servers  (500 out of 4,000) experienced hardware failure. 50 remained offline after almost a week. Customers were understandably irate. What upset them most, according to eWeek, was Hostway's delay in updating them on its progress. But what should have bothered them more, I think, was the indivisibility of their data from individual servers.

Servers die. Even when they don't, each can only occupy one physical location at a time. And many reasons might render any particular location unsuitable: natural disasters, escalating electricity rates, mergers between data center providers...  Server migrations, in other words, are not unlikely occurrences. If you've been in the web hosting business for any length of time, you've almost certainly encountered situations like Hostway's - even if not on the same scale. So no schadenfreude, please.

The moral of Hostway's story is NOT the need for better planning/more proactive communication. Instead, at an age when 86% of CIOs are interested in SaaS and the volume of ecommerce transactions exceeds $100 billion per year, I think hosting providers should be able to relocate/replace hardware without taking the contents of servers offline.

Imagine if Hostway had moved customer apps and data on its failed machines separately from the actual boxes. Hostway operates tens of thousands of servers around the world. Its other data centers quite likely have more than enough capacity to temporarily accommodate the bandwidth/storage/CPU requirements of its Miami customers while their servers were en route. Such a feat can easily be accomplished today. Just ask Albert Wu, who had this to say after moving his entire application across the continent:

"The migration turned out to be an extremely simple process that basically involved issuing a migrate command... Once completed, all we had to do was reconfigure the app with new IP addresses, which only took a few minutes. Amazingly, we were then able to fire up our app! No hardware issues to deal with what-so-ever!"

Or talk to former Netsuite CIO Dave Durkee, who envisions a future in which applications will auto-detect network stability/resource availability and self-migrate to the most suitable hosting facilities. That future is very nearly here, and I hope that Hostway - as well as you, whether you're a web hosting provider or customer - will be a part of it.

Disclaimer: both Albert and Dave are users of 3Tera's utility computing software, while I'm a member of its advisory board.

Comments

[Ditlev Bredahl]

We had to move 2500 servers from Redbus SOV to Interxion (London) a few months ago. It was an absolute horrible experience resulting in 4-6 hours downtime.

We expected 7% of the servers to have hardware issues, but ended up with less than 5%. About 500 of the servers were good old Raq3/4's that had been running for up to 5 years without a reboot.
We had 6 clients leaving us as a direct result of the (announced) downtime.

Moving all applications/content/OS's etc on to new hardware would simply not be doable for us. We would need root psw to all the servers and explicit permission to move content, with the risk of breaking it in the process.
At the same time we would need an army of admins to actually carry out the work - admins that would have sufficient knowledge within all the apps that our clients are running on those boxes.

The move itself was nerve wrecking enough (and quite expensive too) - dealing with clients applications would have made it impossible...

:)
Ditlev, UK2.net

[Isabel Wang]

Hi Ditlev,

But you wouldn't need to have knowledge of individual client apps! On a virtualized platform, you'd be able to move contents of servers between physical machines without knowing what's running on each :)

I love Cobalt RaQs; they're beautiful. Unfortunately no hardware lasts forever. In which case customers will have to migrate code/data to different machines and oftentimes face headaches such as drive incompatibilities. In a virtualized environment, this wouldn't be the case either. They could just copy and paste whatever they had and get up and running instantly.

[Ditlev Bredahl]

true, it would be a different case if we (or hostway) were running a virtualized platform :)

D

[Hosed by Hoseway]

The real moral is that when you plan something this large in scope, you share your grand plans with your paying customers whom it will effect. There was nothing about the move posted on their website, no letters were sent, no phone calls. They said the "emailed" everyone. I didn't get one. And besides, you don't just drop someone an email when you're planning to put someones $5000 server on a truck (I was co-located) and move it somewhere. Then when it takes longer that planned, shut down communication all together, don't answer phone calls, and respond to email tickets 10 days later. That's the Hoseway way!