Peiter "Mudge" Zatko and Sarah Zatko

A 'Consumer Reports' for software vulnerabilities

For years, organizations have struggled with understanding the potential weaknesses of the software they’re using, in part because there is no unbiased measure of comparison to help guide their choices.

At the recent Black Hat conference, long-time computer scientists Peiter 'Mudge' and Sarah Zatko discussed the Cyber Independent Testing Lab, an independent organization to benchmark commercial software security flaws. (Photo by Richard C. Hoffman)

But with the help of one emerging government-supported nonprofit, companies and government agencies alike may soon have a better handle on how software they own or are considering purchasing measures up in terms of security. During a presentation at last week’s Black Hat conference in Las Vegas, computer scientists Peiter Zatko (better known as Mudge) and Sarah Zatko discussed the independent organization they are building to impartially benchmark commercial software security flaws.

“All the certifications and evaluations that come out, they’re not about security,” said Sarah Zatko, who is chief scientist for the Cyber Independent Testing Lab and a member of the Army's Order of Thor, which recognizes contributions of cybersecurity professionals.

Meanwhile, more arcane and technical source code reviews do not help the average corporate or independent software user understand or evaluate the potential security flaws in their software. “Legislation is well-meaning,” she added, “but it typically focuses on making it illegal to look at this problem, and that is a terrible way to solve anything.”

Mudge, a longtime hacker and vulnerability specialist, left Google last year to launch and become the director of CITL after he received a call from the White House urging him to do so. A former program manager at the Defense Advanced Research Projects Agency and author of the password-cracking L0phtCrack software, Mudge is no stranger to the public- and private-sector struggles involved with evaluating the security of the software that organizations are using.

Although they are not planning to certify or offer any seals of approval on the software they test, the Zatkos said CITL aims to use a Consumer Reports-like methodology to evaluate the security of commercial software based on metrics and measurements that will allow laypeople to quantitatively compare different products.

In the year since they began their efforts, the pair and their team at CITL have been using a range of heurisitics that attackers typically use to determine whether software targets are hard or soft, meaning difficult or easy to break into with their exploits. Their metrics and testing are a combination of popular, widely known techniques and “esoteric tradecraft,” and they test software on the three most popular operating systems. So far, the lab has tested software quality and inherent vulnerabilities in more than 100,000 binary applications across Windows, Linux and OS X platforms.

So far, their work has confirmed some basic beliefs about the security of some products, particularly when run on particular operating systems, but it has also challenged conventional wisdom in some areas. Sometimes, CITL found, “the more secure product is actually ... cheaper, and quite often the [expensive] security product is the most vulnerable.”

It’s not surprising, as Mudge pointed out, that the “price tags for exploits mapped up nicely” with how hardened a product is – meaning exploits for hardened software are more expensive. “If we do this measurement for the office suites that are available,” he said, “you start to realize the level of effort an adversary has to go through.”

Mudge also noted the Microsoft Office Suite running on OS X is much softer (and open to compromise) than the same suite running on Windows, because the security Microsoft offers for its own platform is better than what is available when its software runs on competing operating systems.

CITL will start releasing its results in early 2017, the co-founders said.

About the Author

Karen Epper Hoffman is a freelance writer based in the Seattle area.


  • Records management: Look beyond the NARA mandates

    Pandemic tests electronic records management

    Between the rush enable more virtual collaboration, stalled digitization of archived records and managing records that reside in datasets, records management executives are sorting through new challenges.

  • boy learning at home (Travelpixs/

    Tucson’s community wireless bridges the digital divide

    The city built cell sites at government-owned facilities such as fire departments and libraries that were already connected to Tucson’s existing fiber backbone.

Stay Connected