New help on testing for common cause of software bugs

NIST releases a tutorial on automated testing of multiple variables

The National Institute of Standards and Technology has developed algorithms for automated testing of the multiple variables in software that can cause security faults, and has released a tutorial for using the tools.

The improper or unexpected interaction of two or more parameters in a piece of software, such as inputs or configuration settings, is a significant cause of security bugs. But testing for these problems has been limited by the cost and complexity of testing the huge number of possible combinations. NIST in 2003 reported that such problems cost the U.S. economy more than $59 billion a year despite the fact that more than half of most software development budgets went toward testing.

Research has shown that in many cases the large majority of such faults, from 89 to 100 percent, are caused by combinations of no more than four variables, and virtually all are caused by no more than six, NIST has reported.

Related coverage:

NIST test puts software analysis tools through their paces

“This finding has important implications for testing because it suggests that testing combinations of parameters can provide highly effective fault detection,” NIST said in the tutorial,“Practical Combinatorial Testing, Special Publication 800-142.”

Testing pairs of variables, although practical, can miss from 10 percent to 40 percent of system bugs, NIST said. But a lack of good algorithms for testing higher numbers of variables at a time has made such testing impracticably expensive, and is not used except for high-assurance software for mission-critical applications.

The Automated Combinatorial Testing for Software program is a cooperative effort by NIST, the Air Force, the University of Texas at Arlington, George Mason University, Utah State University, the University of Maryland and North Carolina State University to produce methods and tools to generate tests for any number of variable combinations. SP 800-142 offers instructions for their use.

The new algorithms and tools make automated testing for relatively small combinations of variables practical, but combinatorial testing is not cost-free. The NIST publication provides information on the costs and practical considerations for each type of testing, and explains tradeoffs and limitations.


About the Author

William Jackson is a Maryland-based freelance writer.


  • Records management: Look beyond the NARA mandates

    Records management is about to get harder

    New collaboration technologies ramped up in the wake of the pandemic have introduced some new challenges.

  • puzzled employee (fizkes/

    Phish Scale: Weighing the threat from email scammers

    The National Institute of Standards and Technology’s Phish Scale quantifies characteristics of phishing emails that are likely to trick users.

Stay Connected

Sign up for our newsletter.

I agree to this site's Privacy Policy.