Man trying to drag giant heavy website What went wrong?

This story has been updated to correct the number of state health care exchanges.

In some ways, the federal website appears to a victim of its own ambition. A large-scale software project pushed out to meet an Oct. 1 deadline (despite failing a test only days before), it has been plagued by problems since its first day. Now it could take weeks, or even months, to completely fix.

President Obama has said he’s “mad” about the problems that frustrated about 98 percent of the first 20 million visitors to the site, and the Health and Human Services Department has called for a "tech surge," which will include members of the Presidential Innovation Fellows program and Verizon on getting the website up and running smoothly.

Fixing the $400 million system will be a big job, because the system itself is huge. The questions HHS will have to face in the meantime include whether it had to be so big and what steps could have eliminated some of these early problems. will hardly be the last large-scale online project government undertakes, and it could be a good source of lessons learned the hard way.

In terms of software development, has two main parts: a front end -- developed by a startup company and other consultants and formerly hosted on the public GitHub library -- and a backend developed by CGI Federal and 54 other contractors, working separately on software that HHS then had to ensure worked together. The problems are on the back end, which was not developed in public.

Along the way in development, the software for that back end grew to a considerable size. In a report over the weekend, the New York Times cited a specialist who estimated that the site could contain about 500 million lines of code and that as many as 5 million lines of code will have to be rewritten. That’s an estimate, and other system designers have posited smaller numbers, but an overabundance of code can slow any system down and cause other problems. The Times noted that a large bank’s computer system is about one-fifth the size of the 500 million-line figure.

Slate reported that such a large stash of code could indicate that contractors “may be writing their own code in many places where they'd be better off relying on open-source external libraries. They may also be solving problems via copy-and-paste rather than more elegant programming techniques such as inheritance or polymorphism.”

Of course, such bloat in a software project also can be caused if an agency changes its requirements often, causing contractors to adjust on the fly. The Times reported that hardware and software requirements on the project had been modified seven times in the last 10 months.

Regardless of the cause, the site wasn’t ready as the deadline approached, but HHS had to stick to the publically declared deadline. The Washington Post reported  that, mere days before the launch date, HHS tested the system’s ability to handle tens of thousands of users at once. It crashed after a few hundred.

A week before the launch, the site hadn’t been tested to see whether a single user could get all the way through the process, the Post reported. And a test group of insurance companies had warned HHS a month earlier not to launch the site because of problems with the system.

Perhaps another thing had working against it is just the fact that it is a large IT project, and large IT projects routinely have troubles and sometimes fail altogether. Computerworld asked the Standish Group, which tracks IT development projects, to examine its list of multimillion-dollar projects to see how they fared. Only 6.4 percent of projects with labor costs of $10 million or more were deemed successful. More than 41 percent were abandoned or started over, and the remaining 52 percent were over budget, behind schedule or fell short of users’ expectations, Computerworld reported.

So maybe a troubled launch for such a large project is to be expected. It’s worth noting that the 17 states  and the District of Columbia that built their own health care exchange sites — performing the same task but on smaller scales — reportedly are working much better than the federal site.

Government is no stranger to large-scale IT projects gone wrong, from the FBI’s $170 million Virtual Case File system to the Air Force’s abandoned, $1 billion enterprise resource planning system. By those standards, if HHS’s tech surge can get the health exchange site in working order — recent reports are that it is slowly improving — might eventually qualify as a success.


  • Records management: Look beyond the NARA mandates

    Pandemic tests electronic records management

    Between the rush enable more virtual collaboration, stalled digitization of archived records and managing records that reside in datasets, records management executives are sorting through new challenges.

  • boy learning at home (Travelpixs/

    Tucson’s community wireless bridges the digital divide

    The city built cell sites at government-owned facilities such as fire departments and libraries that were already connected to Tucson’s existing fiber backbone.

Stay Connected