CyberEye

Blog archive
HealthCaregov test checklist

Lesson from HealthCare.gov: A launch is no time for a beta test

So, just how bad was the roll-out of the Affordable Care Act portal, HealthCare.gov?  It depends.

As a production launch, it was really bad. Failures on the site frustrated users and burned up a lot of goodwill that the administration could ill afford to lose.

Other views

Media got it wrong: HealthCare.gov failed despite agile practices

Developer documentation shows agile development practices were used in building the site. Here's where that could have hurt. Read more.

Why Agile can work for complex systems like HeathCare.gov

Development of large, complex systems requires a method that also allows a progressive discovery of project scope and dependencies, along with identification and mitigation of key risks. Read more.

As a beta test, it still was pretty bad; but that’s what beta tests are for. There is no disgrace in finding problems in an application during testing. “This is pretty standard,” said David Lindsay, senior security product manager at Coverity, a development testing company. The failure with HealthCare.gov is that the beta test started Oct. 1 rather than two months earlier.

Any large application project requires testing throughout the planning and development phases. This begins with the individual components as they are developed and continues as they are assembled into working parts. “There is still going to be a need for production testing,” when the entire application is completed, Lindsay said. “A final pass before things go out the door.”

There is a paradox in large project developments, however. The more complex the application, the more testing it needs. But the more complex it is, the later all of the parts come together and the less time there is for final testing. And it is not enough to squirt bits at the software on a laboratory testbed. It should be exposed to real users who can put it through its paces to expose the unintended consequences of development decisions. This is the beta test, the last stop before production release and the last chance to fix problems before you disappoint users and delight antagonists.

HealthCare.gov certainly qualifies as a complex application. It serves as a gateway for 36 state programs and must securely access sensitive information in multiple state and federal databases. The results will determine whether and what kind of health care millions of people have access to, so the stakes are high.

Doing a beta test for a large-scale public Web application is not simple, but it can be quietly and slowly rolled out in a number of locations. Given the current projected fix date of Nov. 30 for the site, this testing phase should have begun no later than Aug. 1 in order to be ready for production by Oct. 1. But the current fix is being done on a crash basis, which is not a good way to develop and fix software. So a July date for beta release would have made better sense.

This would not have been simple to do. All of the states involved and the insurance companies offering coverage would have needed to have their parts ready three months early. That kind of scheduling has to be built into the project from the beginning, not brought up as an afterthought. Synchronizing the release date with the launch date is a recipe for failure, and that is what happened with HealthCare.gov.

There was time to do it right. The Affordable Care Act was passed in early 2010. The task was complicated, however, by opponents of the program who helped to talk many states out of offering their own healthcare exchanges, making the federal portal that much more complex. Those opponents share much of the blame for the botched launch of the site. But that does not excuse those who planned and developed the system from doing it right.

Posted by William Jackson on Nov 01, 2013 at 6:20 AM


Reader Comments

Fri, Dec 20, 2013 Maria

I was in agreement with this article until I came to the last paragraph: "opponents of the program who helped to talk many states out of offering their own healthcare exchanges, making the federal portal that much more complex. Those opponents share much of the blame for the botched launch of the site." Really? There were 3 options for running exchanges for States to choose from: Total Federal, Federal/State, and Total State. With so many States struggling financially, deciding to let the Fed Government build and pick up the tab was a no-brainer. And since it was an option, the Feds should have prepared for that possibility and that extra "complexity". States are free to make whatever choice best met that State's needs. Sorry, but opponents don't share the blame for this "crash and burn" and it was disingenuous to suggest otherwise.

Wed, Dec 4, 2013 GA

I wish I could blame "opponents" for crap code whenever I wrote a bug.

Thu, Nov 21, 2013

Get real! The Government launches web based stuff more times than anyone can imagine just like the healthcare thing.

Tue, Nov 5, 2013 Don O'Neill

Is the ACA web site still working off Technical Debt? A Beta Test is an exercise to demonstrate that a system work as intended, not an exercise to detect defects tom pollute Jeffrey Zients's punch list. The question remains, will the November 30 ACA web site be a system that has completed Beta Test or one that has not yet entered Beta Test?

Mon, Nov 4, 2013

WIth emerging technologies such Service Virtualization CMS and their supporting contractors could have conducted end to end performance testing MONTHS earlier and been better prepared for a successful launch even with the late requirements that did not emerge until Spring, '13

Show All Comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

resources

HTML - No Current Item Deck
  • Transforming Constituent Services with Business Process Management
  • Improving Performance in Hybrid Clouds
  • Data Center Consolidation & Energy Efficiency in Federal Facilities