VARs: Disaster Recovery Plans Reduced Hurricane Irene's Impact

Solution providers and their customers are still adding up the damage caused when Hurricane Irene slammed the East Coast this weekend, but they said that advanced disaster preparation meant IT problems caused by the storm were much lighter than they would have been in the past.

A combination of preparation for potential disasters long before Irene was spawned, proactive contact in the days before Irene hit the coast, and a move to outsource many IT functions meant IT operations along the Atlantic Seaboard escaped the kind of damage caused by hurricanes such as Hurricane Katrina, which hit the Gulf of Mexico states in 2005, and Isabel, which soaked the East Coast in 2003.

For Peter Katz, president of PKA Technologies, a Suffern, N.Y.-based solution provider, the experience of one customer, cut off from its offices, forced PKA to immediately decide to move to the cloud to protect itself from future disasters.

That customer found its facilities completely operational after the hurricane once its backup generators kicked in, but water prevented personnel from getting inside, Katz said.

"They couldn't get to the office because it was surrounded by water," he said. "The police wouldn't let them go in. They would need a boat to get there. I would never have thought for one second about not having power in my executive center. But all of a sudden, I saw what happened to the customer. We have to change."

Katz said the experience of that customer worried hime and his staff over the weekend about what would happen if PKA's own offices would be unavailable for a week or two.

"So over the weekend we decided to move our entire infrastructure to the cloud," Katz said. "Our quoting system, our ability to update the status of customer orders, all customer-facing applications, we'll move them to the cloud. Within a year from now, I'll be so remotely-enabled that we won't have to worry about running out of electricity."

Hurricane Irene helped drive home many lessons related to disaster recovery, said Michael Oh, founder and president of Tech Superpowers, a Boston-based solution provider.

Fortunately for Tech Superpowers' customers, there were no real disasters, Oh said. "However, clients suddenly realized the benefit of offsite backup," he said.

One customer lost power for a short time, causing its network to crash, but all the servers remained in operation except the one which was not connected to a UPS, Oh said.

The only real problem Tech Superpowers had was one remote employee who lost power at home and struggled to find a place from which to work, Oh said.

"He went to all the Starbucks in the area, but the power was down at all of them," he said. But he eventually wound up working at Panera, a local bakery chain, where the network was running. The place was full of people with their laptops."

Oh said one major lesson of the hurricane was the potential geographic impact such a disaster caused.

Many of Tech Superpowers' customers had their servers and services hosted by Hosted Solutions, a service provider acquired late last year by managed hosted services provider Windstream.

Next: More Lessons Learned

As a result, Oh said, customers whose primary operations were hosted at Hosted Solutions had those operations backed up by Windstream's facilities in Raleigh, N.C., which was also hit hard by Hurricane Irene.

"Under no circumstances would we expect anything to impact both facilities," he said. "No terror attack would do it. Maybe a nuclear blast. But luckily for customers, both facilities remained online."

Hurricane Irene caused Tech Superpowers to ask itself whether it selected the right partner and did enough to protect customers against a disaster, Oh said.

"For the first question, the answer is yes," he said. "Our provider had not problems. It stayed online. For the second question, however, we need to look at disaster plans in terms of geography."

Eryck Bredy, founder and CTO of BNMC, an Andover, Mass.-based solution provider who also worked with Windstream/Hosted Solutions as a partner, said customers were relieved that they didn't lose any data during the hurricane.

"Some customers worried because our backup data center was in Raleigh, N.C.," Bredy said. "But we told customers their data was stored in two places, so it was safe."

BNMC proactively e-mailed customers before Irene arrived to remind them that the company's personnel would be available around the clock over the weekend should they need any help, a move Bredy said helped diffuse any possible anxieties customer might have.

"We were over-prepared," he said. In the Long Island area of New York, winds and flooding killed the power to HorizonTek, a Huntington, N.Y.-based storage solution provider, but otherwise did not hurt the company's facilities, said Chris Leonard, a sales executive.

HorizonTek used the home of one of its executives as a backup data center, and was able to continue operations on a remote basis with no major issues, Leonard said.

For Doug Cole, a partner at LH Computer Services, a Coral Springs, Fla.-based solution provider, Hurricane Irene was unusual in that it did not have much impact locally, but it did cause some power outages for customers up the Atlantic coast all the way up to Boston.

Preparing for possible disasters save customers, Cole said.

"Nobody lost any data," he said. "We started contacting them a week ago to ask if they needed any help. They were busy hunkering down, getting ready to execute their disaster recovery plans. Not make disaster recovery plans, but execute on them."

Cole said he would like to receive the credit for saving customers from disaster. "But a lot of the solutions we put in include SANs and replication, and all include good backups," he said. "Disaster recovery is a part of our solutions. So we have no big, sad stories to tell."

Some places were hit harder than others.

Hope Hayes, president of Alliance Technology Group, a Hanover, Md.-based solution provider, said her building received only a small percentage of the power it needs to run on Monday, and that half her employees' homes are without power, which means it was difficult to contact customers.

The disaster struck just as Alliance was in the process of finalizing negotiations on a new building in which a state-of-the-art data center is being planned, Hayes said. However, because of the hurricane, Alliance will have to invest in fail-over technology in its current space, which is already overcrowded because of growth over the last few years.

"We're moving," she said. "We planned to put in all the failover technology in the new office. But now we can't wait."

Next: Not Like Past Disasters

The emphasis customers are placing on being prepared for a disaster is completely difference from attitudes in the past.

Katz said that, immediately after the September 11, 2001 terrorist attacks, the biggest topic among customers was disaster recovery and disaster tolerance.

"After the attacks, I went and talked to customers about it," he said. "How many sales did I make? Hello? It's like teenage sex. Everybody talks about it, but no one is doing it."