New help on testing for common cause of software bugs

NIST releases a tutorial on automated testing of multiple variables

The National Institute of Standards and Technology has developed algorithms for automated testing of the multiple variables in software that can cause security faults, and has released a tutorial for using the tools.

The improper or unexpected interaction of two or more parameters in a piece of software, such as inputs or configuration settings, is a significant cause of security bugs. But testing for these problems has been limited by the cost and complexity of testing the huge number of possible combinations. NIST in 2003 reported that such problems cost the U.S. economy more than $59 billion a year despite the fact that more than half of most software development budgets went toward testing.

Research has shown that in many cases the large majority of such faults, from 89 to 100 percent, are caused by combinations of no more than four variables, and virtually all are caused by no more than six, NIST has reported.

Related coverage:

NIST test puts software analysis tools through their paces

“This finding has important implications for testing because it suggests that testing combinations of parameters can provide highly effective fault detection,” NIST said in the tutorial,“Practical Combinatorial Testing, Special Publication 800-142.”

Testing pairs of variables, although practical, can miss from 10 percent to 40 percent of system bugs, NIST said. But a lack of good algorithms for testing higher numbers of variables at a time has made such testing impracticably expensive, and is not used except for high-assurance software for mission-critical applications.

The Automated Combinatorial Testing for Software program is a cooperative effort by NIST, the Air Force, the University of Texas at Arlington, George Mason University, Utah State University, the University of Maryland and North Carolina State University to produce methods and tools to generate tests for any number of variable combinations. SP 800-142 offers instructions for their use.

The new algorithms and tools make automated testing for relatively small combinations of variables practical, but combinatorial testing is not cost-free. The NIST publication provides information on the costs and practical considerations for each type of testing, and explains tradeoffs and limitations.


About the Author

William Jackson is a Maryland-based freelance writer.


  • business meeting (Monkey Business Images/

    Civic tech volunteers help states with legacy systems

    As COVID-19 exposed vulnerabilities in state and local government IT systems, the newly formed U.S. Digital Response stepped in to help. Its successes offer insight into existing barriers and the future of the civic tech movement.

  • data analytics (

    More visible data helps drive DOD decision-making

    CDOs in the Defense Department are opening up their data to take advantage of artificial intelligence and machine learning tools that help surface insights and improve decision-making.

Stay Connected