New help on testing for common cause of software bugs

NIST releases a tutorial on automated testing of multiple variables

The National Institute of Standards and Technology has developed algorithms for automated testing of the multiple variables in software that can cause security faults, and has released a tutorial for using the tools.

The improper or unexpected interaction of two or more parameters in a piece of software, such as inputs or configuration settings, is a significant cause of security bugs. But testing for these problems has been limited by the cost and complexity of testing the huge number of possible combinations. NIST in 2003 reported that such problems cost the U.S. economy more than $59 billion a year despite the fact that more than half of most software development budgets went toward testing.

Research has shown that in many cases the large majority of such faults, from 89 to 100 percent, are caused by combinations of no more than four variables, and virtually all are caused by no more than six, NIST has reported.

Related coverage:

NIST test puts software analysis tools through their paces

“This finding has important implications for testing because it suggests that testing combinations of parameters can provide highly effective fault detection,” NIST said in the tutorial,“Practical Combinatorial Testing, Special Publication 800-142.”

Testing pairs of variables, although practical, can miss from 10 percent to 40 percent of system bugs, NIST said. But a lack of good algorithms for testing higher numbers of variables at a time has made such testing impracticably expensive, and is not used except for high-assurance software for mission-critical applications.

The Automated Combinatorial Testing for Software program is a cooperative effort by NIST, the Air Force, the University of Texas at Arlington, George Mason University, Utah State University, the University of Maryland and North Carolina State University to produce methods and tools to generate tests for any number of variable combinations. SP 800-142 offers instructions for their use.

The new algorithms and tools make automated testing for relatively small combinations of variables practical, but combinatorial testing is not cost-free. The NIST publication provides information on the costs and practical considerations for each type of testing, and explains tradeoffs and limitations.


About the Author

William Jackson is a Maryland-based freelance writer.


  • Records management: Look beyond the NARA mandates

    Pandemic tests electronic records management

    Between the rush enable more virtual collaboration, stalled digitization of archived records and managing records that reside in datasets, records management executives are sorting through new challenges.

  • boy learning at home (Travelpixs/

    Tucson’s community wireless bridges the digital divide

    The city built cell sites at government-owned facilities such as fire departments and libraries that were already connected to Tucson’s existing fiber backbone.

Stay Connected