Man trying to drag giant heavy website

HealthCare.gov: What went wrong?

This story has been updated to correct the number of state health care exchanges.

In some ways, the federal HealthCare.gov website appears to a victim of its own ambition. A large-scale software project pushed out to meet an Oct. 1 deadline (despite failing a test only days before), it has been plagued by problems since its first day. Now it could take weeks, or even months, to completely fix.

President Obama has said he’s “mad” about the problems that frustrated about 98 percent of the first 20 million visitors to the site, and the Health and Human Services Department has called for a "tech surge," which will include members of the Presidential Innovation Fellows program and Verizon on getting the website up and running smoothly.

Fixing the $400 million system will be a big job, because the system itself is huge. The questions HHS will have to face in the meantime include whether it had to be so big and what steps could have eliminated some of these early problems.

HealthCare.gov will hardly be the last large-scale online project government undertakes, and it could be a good source of lessons learned the hard way.

In terms of software development, HealthCare.gov has two main parts: a front end -- developed by a startup company and other consultants and formerly hosted on the public GitHub library -- and a backend developed by CGI Federal and 54 other contractors, working separately on software that HHS then had to ensure worked together. The problems are on the back end, which was not developed in public.

Along the way in development, the software for that back end grew to a considerable size. In a report over the weekend, the New York Times cited a specialist who estimated that the site could contain about 500 million lines of code and that as many as 5 million lines of code will have to be rewritten. That’s an estimate, and other system designers have posited smaller numbers, but an overabundance of code can slow any system down and cause other problems. The Times noted that a large bank’s computer system is about one-fifth the size of the 500 million-line figure.

Slate reported that such a large stash of code could indicate that contractors “may be writing their own code in many places where they'd be better off relying on open-source external libraries. They may also be solving problems via copy-and-paste rather than more elegant programming techniques such as inheritance or polymorphism.”

Of course, such bloat in a software project also can be caused if an agency changes its requirements often, causing contractors to adjust on the fly. The Times reported that hardware and software requirements on the project had been modified seven times in the last 10 months.

Regardless of the cause, the site wasn’t ready as the deadline approached, but HHS had to stick to the publically declared deadline. The Washington Post reported  that, mere days before the launch date, HHS tested the system’s ability to handle tens of thousands of users at once. It crashed after a few hundred.

A week before the launch, the site hadn’t been tested to see whether a single user could get all the way through the process, the Post reported. And a test group of insurance companies had warned HHS a month earlier not to launch the site because of problems with the system.

Perhaps another thing HealthCare.gov had working against it is just the fact that it is a large IT project, and large IT projects routinely have troubles and sometimes fail altogether. Computerworld asked the Standish Group, which tracks IT development projects, to examine its list of multimillion-dollar projects to see how they fared. Only 6.4 percent of projects with labor costs of $10 million or more were deemed successful. More than 41 percent were abandoned or started over, and the remaining 52 percent were over budget, behind schedule or fell short of users’ expectations, Computerworld reported.

So maybe a troubled launch for such a large project is to be expected. It’s worth noting that the 17 states  and the District of Columbia that built their own health care exchange sites — performing the same task but on smaller scales — reportedly are working much better than the federal site.

Government is no stranger to large-scale IT projects gone wrong, from the FBI’s $170 million Virtual Case File system to the Air Force’s abandoned, $1 billion enterprise resource planning system. By those standards, if HHS’s tech surge can get the health exchange site in working order — recent reports are that it is slowly improving — HealthCare.gov might eventually qualify as a success.

Reader Comments

Thu, Oct 31, 2013 Cowboy Joe

Gotta agree with AG2013, with this caveat. There's plenty of tools and methodologies that could have worked, "agile" being only one of 'em. Agree with the whole ACA premise or not, the crux of the failure was that nobody really agreed ahead of time how all the pieces should work together ... or even that they should work together at all.

Wed, Oct 30, 2013 AG2013

Agile zealots are quick to put the blame fully on the project not having used Agile methodology. This is pure hogwash. Had Healthcare.gov used Agile, it would still be a miserable failure. The UK's massive "Universal Credit" project was an Agile disaster for the same reason. The real reason why Healthcare.gov failed is because it's a very complex system that's designed, administered, and programmed by a disorganized, bureaucratic team.

Wed, Oct 23, 2013 MikeC

There is plenty of blame for all parties here. Developing a deliverable with 50+ companies involved ... takes a lot more skill than our federal employees have. No agile methodology, testing, delivery, stakeholders ... very 1980s approach to a 2013 problem. Spec changes are expected, but sounds like there were too many. Vendor selection likely very bad. Politics before technology (ie: prices so high that you could not "shop" until you created your account ... bad paradigm. Shopping should be light, account creation (as we've seen) is heavy. Fed should have done it very differently from the onset.

Wed, Oct 23, 2013

At the KFF link tied to statement the "14 states and the Distric" have exchanges, where their "Data are as of May 28, 2013" shows 17 states as SBM/SBE, not 15. what is the right number?

Wed, Oct 23, 2013

I agree that the site is poorly architected as well as poorly planned. I have a theory that the site was built by a tea-party republican who's against the Obama health care. It's just another conspiracy against Obama's health care reform.

Show All Comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above