HealthCare.gov: What went wrong?
Connecting state and local government leaders
The federal health care exchange website was forced into action by a hard deadline, despite bloated code and failed tests. What can agencies learn from the launch?
This story has been updated to correct the number of state health care exchanges.
In some ways, the federal HealthCare.gov website appears to a victim of its own ambition. A large-scale software project pushed out to meet an Oct. 1 deadline (despite failing a test only days before), it has been plagued by problems since its first day. Now it could take weeks, or even months, to completely fix.
President Obama has said he’s “mad” about the problems that frustrated about 98 percent of the first 20 million visitors to the site, and the Health and Human Services Department has called for a "tech surge," which will include members of the Presidential Innovation Fellows program and Verizon on getting the website up and running smoothly.
Fixing the $400 million system will be a big job, because the system itself is huge. The questions HHS will have to face in the meantime include whether it had to be so big and what steps could have eliminated some of these early problems.
HealthCare.gov will hardly be the last large-scale online project government undertakes, and it could be a good source of lessons learned the hard way.
In terms of software development, HealthCare.gov has two main parts: a front end -- developed by a startup company and other consultants and formerly hosted on the public GitHub library -- and a backend developed by CGI Federal and 54 other contractors, working separately on software that HHS then had to ensure worked together. The problems are on the back end, which was not developed in public.
Along the way in development, the software for that back end grew to a considerable size. In a report over the weekend, the New York Times cited a specialist who estimated that the site could contain about 500 million lines of code and that as many as 5 million lines of code will have to be rewritten. That’s an estimate, and other system designers have posited smaller numbers, but an overabundance of code can slow any system down and cause other problems. The Times noted that a large bank’s computer system is about one-fifth the size of the 500 million-line figure.
Slate reported that such a large stash of code could indicate that contractors “may be writing their own code in many places where they'd be better off relying on open-source external libraries. They may also be solving problems via copy-and-paste rather than more elegant programming techniques such as inheritance or polymorphism.”
Of course, such bloat in a software project also can be caused if an agency changes its requirements often, causing contractors to adjust on the fly. The Times reported that hardware and software requirements on the project had been modified seven times in the last 10 months.
Regardless of the cause, the site wasn’t ready as the deadline approached, but HHS had to stick to the publically declared deadline. The Washington Post reported that, mere days before the launch date, HHS tested the system’s ability to handle tens of thousands of users at once. It crashed after a few hundred.
A week before the launch, the site hadn’t been tested to see whether a single user could get all the way through the process, the Post reported. And a test group of insurance companies had warned HHS a month earlier not to launch the site because of problems with the system.
Perhaps another thing HealthCare.gov had working against it is just the fact that it is a large IT project, and large IT projects routinely have troubles and sometimes fail altogether. Computerworld asked the Standish Group, which tracks IT development projects, to examine its list of multimillion-dollar projects to see how they fared. Only 6.4 percent of projects with labor costs of $10 million or more were deemed successful. More than 41 percent were abandoned or started over, and the remaining 52 percent were over budget, behind schedule or fell short of users’ expectations, Computerworld reported.
So maybe a troubled launch for such a large project is to be expected. It’s worth noting that the 17 states and the District of Columbia that built their own health care exchange sites — performing the same task but on smaller scales — reportedly are working much better than the federal site.
Government is no stranger to large-scale IT projects gone wrong, from the FBI’s $170 million Virtual Case File system to the Air Force’s abandoned, $1 billion enterprise resource planning system. By those standards, if HHS’s tech surge can get the health exchange site in working order — recent reports are that it is slowly improving — HealthCare.gov might eventually qualify as a success.
NEXT STORY: New phone on the block