Obamacare Site Disaster: 10 Steps Solution Providers Would Take To Fix It

Fixing A Blunder

As the government attempts to figure out the problems with the Obamacare flagship site and the backend databases, solution providers said that there are options for a few so-called "quick wins" to help the site get up and rolling. The 20 million hits on the website show that people are "voting with their mouse clicks," said David Lloyd, CEO of IntelliResponse, and that many people want to use the site to sign up rather than phone calls or paperwork.

In light of all the technical problems, CRN asked solution providers what they would have done if put in charge of the site and what they think the best course is going forward. Here's a look at the first 10 first steps solution providers say they would take to get the site working again.

10. Too Much, Too Soon

The first step that should have been taken was to follow best practices from the beginning, solution providers told CRN. "If you build performance culture early on, ... finding the problems as you go along is a lot more efficient than trying to find them at the end of the process," said Mark Burns, director of product management at Compuware APM. If best practices had been followed, the time constraints on the launch would not have been as much of an issue, Burns said.

Robin Purohit, CEO of Clustrix, which helps support medical information database creator MedExpert, said that his company has found it is best practice to develop one function at a time, both to test individual performance and communicate the results to the customer. However, a lot of people still follow the "waterfall" approach, he said, which isn't nearly as effective. Purohit said that he has seen startups go from zero to 10 million users in three months with no problems when using best practices. "This can be done," Purohit said.

9. Do A Scaled Release

Instead of launching the site to the entire country all at once, it would have been better to do a scaled rollout, said David Lloyd, CEO of IntelliResponse. He compared the situation to a Black Friday rush, where it's better to let customers slowly come onto the site rather than allowing a stampede of holiday shoppers trample each other for the latest toy.

He suggested first offering the program to small businesses as a smaller-scale trial run and implementing the lessons learned from that rollout before expanding the offering to individuals and families.

"They could have scaled it much differently than going for the big bang theory of providing the service," Lloyd said.

8. Use More Cloud, Virtualization

One of the major areas of blame for the healthcare.gov site fiasco was the immense amount of traffic that hit the site since its Oct. 1 launch, already logging more than 20 million hits. The problem could have been helped if the developers used a scalable cloud approach instead of throwing more money into hardware, Clustrix CEO Purohit said. The hardware would also work, but it is a lot more labor and capital intensive, he said.

"It's unbelievable what you could do now with the cloud approach," Purohit said. "The cloud is the best arch to deal with unpredictable workloads and data sizes."

Skyler Ferran, COO at HillSouth, said to just look at Netflix, which his company works with. Netflix traffic spikes as people come home from work, so the site keeps extra space on standby to accommodate the additional traffic. Although cloud virtualization becomes trickier with personalization, Ferran said the same approach could have helped the healthcare.gov site.

7. Standardize The Databases

Even though Connecticut's state-run health exchange has been running well, with no downtime so far, interacting with the federal database system is still a struggle, Kevin Counihan, CEO of the Connecticut Health Care Exchange.

The site has to interact with a variety of databases from insurance companies, healthcare organizations and the IRS, to name a few. There isn't a set of standards among the databases, which HillSouth's Ferran expects is one of the problems with the healthcare exchange. "It's almost like they're reinventing the wheel. There are solutions out there already," Ferran said.

One solution would be to force healthcare providers to standardize their databases instead of simply adhering to what they had been doing before," said IntelliResponse's Lloyd. "You end up doing the same thing a hundred different ways with each provider," Lloyd said.

6. Improve Search Capabilities, Add Virtual Agent

After experimenting on the site, IntelliResponse CEO Lloyd said it needs improved search capabilities and a virtual agent to assist consumers. Instead of clicking all over the site, which chews up bandwidth, consumers could find their answers much more quickly, Lloyd said. For example, he said, his own experiment showed that if he searched "I have a heart condition" or "What if I live in California," which has a state exchange, no results popped up. But, if you type into the search that you cannot afford insurance, for example, it spews out more than 100 results that aren't necessarily related.

He said the site is preventing people from self-educating, which 20 million-plus site clicks proved people wanted to do. Going forward, Lloyd said, the search data would help refine the site in the future, as companies like Amazon and Google already do. "That should help them understand how the healthcare.gov has to transform itself to actually support what the consumers are looking for," Lloyd said.

5. Have A Functions Road Map

One of the keys to the Connecticut state health exchange was a tough decision in January to cut functionality by 30 to 35 percent, said exchange CEO Counihan. "In retrospect I think it was the smartest decision we made," Counihan said.

Clustrix's Purohit said he suspected that part of the problem with the site was discussions about what functionalities should be included. "I could only imagine that all the internal bureaucratic debates on what the app should do and shouldn't do ... was a big part of the problem," Purohit said.

Enhancing that issue was the contract basis of the work, said IntelliResponse CEO Lloyd, because workers are often paid per line of code. He joked that it makes people more inclined to code Ferrari-like features into the system rather than taking a simplified business approach.

4. Minimize Third-Party Components

Using its external analysis software, Compuware APM did a thorough analysis of the healthcare.gov site. While it could not see all the back-end systems without access, it was able to diagnose many site problems. "There were a lot of extras," Compuware's Burns said, which created inefficiency. "It's like everything and the kitchen sink was left in the website whether it was needed or not."

One major problem was the "broad overuse" of third-party services, a number of which were likely unnecessary because many of the third parties performed the same tasks, Burns said. Financial services companies that Compuware works with, he said, usually have five to seven third parties involved with their sites. The healthcare.gov site had well over 20 or 30 third parties, Burns said the analysis found, which he called a "very aggressive overuse of these third-party services."

The other main problem with the third parties is that they are out of the site's control, Burns said. If the third party can't handle the traffic load, then the site will collapse.

3. Repeat Calling

Basic performance culture is to eliminate the amount of times that the site must call up Web elements. According to Compuware APM's analysis of the site, it takes an average of more than 16 seconds for each time the site has to call up a page.

"Each one is a performance penalty to the website," said Compuware APM's Burns. "It's like doing laps on the track. Four versus 100, ... it slows things down. It takes a lot longer for the website to load."

HillSouth's Ferran said it would make a lot more sense for the government to pull in companies like Facebook or Twitter, both of which have millions of users pulling information from different sources all of the time.

2. Application Performance Management

Multiple companies have offered their application performance management software to the healthcare site, including Compuware APM and AppDynamics. Compuware APM has already done an external analysis of the site, but it would be helpful to do analysis to discover the location of bottlenecks, Compuware's Burns said.

"Without having a system that can do this accurate performance analysis, they're really falling into the age-old trap of looking in all the areas without having any visibility of what to fix," Burns said. "Without that, they're really flying blind."

He said the government is bringing in some great companies to help fix the problems, such as Verizon, but without APM, the companies won't have any idea where to look.

"I think they're bringing in some really good people who could help if the problem is in a specific area, but I think they're really guessing at this point," Burns said.

1. Test, Test, Test

At the congressional hearing Thursday with the major contractors involved in building the healthcare.gov site, none of the contractor representatives were able to answer the question of who was in charge of compiling the pieces and testing the final project, except for generally attributing it to CMS. Testing is usually the last piece of the project, but it is "crucially important," Compuware's Burns said. "I think the biggest problem is that, especially when you're trying to get the website to market, if you don't have a focus on performance, ... it's often too late to cross your fingers and hope performance is good," Burns said. "We recommend performance testing before a website goes live."

Connecticut Health Care Exchange's Counihan said the Exchange tested its site a few months before launch and found 67 defects, which it was able to fix before the launch.

In the end, problems after rollout damage the company's brand, Burns said, adding that he expects there to be a lingering reputational impact on the site going forward because of the botched implementation.