fingerprints

Approximate matching can help find needles in haystacks

Finding malicious code is not too difficult if you have a fingerprint or signature to look for. Traditional signature-based antivirus tools have been doing this effectively for years. But malware often morphs, adapts and evolves to hide itself, and a simple one-to-one match no longer is adequate.

The National Institute of Standards and Technology is developing guidance for a technique called approximate matching to help automate the task of identifying suspicious code that otherwise would fall to human analysts. The draft document is based on work of NIST’s Approximate Matching Working Group.

“Approximate matching is a promising technology designed to identify similarities between two digital artifacts,” the draft of Special Publication 800-168 says. “It is used to find objects that resemble each other or to find objects that are contained in another object.” 

The technology can be used to filter data for security monitoring and for digital forensics, when analysts are trying to spot potential bad actors either before or after a security incident.

Approximate matching is a generic term describing any method for automating the search for similarities between two digital artifacts or objects. An “object” is an “arbitrary byte sequence, such as a file, which has some meaningful interpretation.”

Humans can understand the concept of similarity intuitively, but defining the aspects of similarity for algorithms can be challenging. In approximate matching, similarity is defined for algorithms in terms of the characteristics of artifacts being examined. These characteristics can include byte sequences, internal syntactic structures or more abstract semantic attributes similar to what human analysts would look for.

Different methods for approximate matching operate at different levels of abstraction. These range from generic techniques at the lowest level to detect common byte sequences, to more abstract analysis that approach the level of human evaluation. “The overall expectation is that lower level methods would be faster, and more generic in their applicability, whereas higher level ones would be more targeted and require more processing,” the document explains.

Approximate matching uses two types of queries: resemblance and containment. Two successive versions of a piece of code are likely to resemble each other, and a resemblance query simply identifies two pieces of code that are substantially similar. With a containment query, two objects of substantially different size, such as a file and a whole-disk image, are examined to determine whether the smaller object, or something similar to it, is contained in the large one.

As described in the document, approximate matching usually is used to filter data, as in blacklisting known malicious artifacts or anything closely resembling them. “However, approximate matching is not nearly as useful when it comes to whitelisting artifacts, as malicious content can often be quite similar to benign content,” NIST warns.

The publication lays out essential requirements of approximate matching functions as well as the factors—including sensitivity and robustness, precision and recall and security—that determine the reliability of the results.

Comments on the publication should be sent by March 21 to match@nist.gov with “Comments on SP 800-168” in the subject.

Posted by William Jackson on Feb 07, 2014 at 10:23 AM1 comments


Internet

Mobile, enterprise users drive US IPv6 growth

According to the latest quarterly State of the Internet report from Akamai, Western nations are leading the way in use of next generation Internet Protocols, with Asia surprisingly lagging behind.

The amount of IPv6 Internet traffic hitting Akamai’s global content distribution network grew sharply in the third quarter of 2013, and the United States and Europe appear to dominate in adoption of the next-generation Internet Protocols.

Only one Asian nation, Japan, was included among the top 10 countries generating IPv6 traffic, with 1.9 percent of its traffic using IPv6. The United States was in fifth place with 4.2 percent. 

“IPv6 uptake in Asia was not as high as we expected it to be,” said David Belson, Akamai’s senior director of industry and data intelligence and lead author of the report. “That was surprising, given the shortage of IPv4 addresses,” in that region.

A limited number of IP addresses are available in Version 4 of the Internet Protocols, and those are beginning to run out. Increasingly, large allocations of addresses are being made from the much larger pool of IPv6 addresses. Because the two versions are not compatible and Internet connected systems have to be readied for the new protocols, many vendors, carriers and infrastructure operators are tracking their adoption closely. 

In the United States, federal agencies are required to accept IPv6 traffic on all public-facing systems. Agencies must upgrade applications that communicate with public Internet servers to use native IPv6 by the end of the 2014 fiscal year. 

The reason for the higher rate of adoption in the Western countries appears to be leadership from mobile carriers as well as government. “It was good for them to put out a deadline” for enabling IPv6 in government systems, Belson said of the U.S. government. But the largest driver is adoption of the protocols by large mobile carriers. Because of quick market growth and a high turnover rate for devices, mobile users are in the forefront of IPv6 adoption, whether they know it or not.

Still, adoption of the new protocols in this country remains spotty. Comcast, the nation’s largest Internet service provider, reports that 25 percent of its customers are provisioned with dual-stack broadband connections supporting IPv6. But consumer hardware such as routers and cable modems tend to stay in place longer than mobile devices, reducing the rate of adoption of the new protocols.

One interesting pattern found by Akamai in IPv6 traffic is that volumes drop each Saturday, meaning that there probably is a higher level of IPv6 adoption on enterprise networks as opposed to consumer ISPs.

Although Internet growth is expected to be in the IPv6 address space, IPv4 is not yet dead. Akamai identified almost 761 million unique IPv4 addresses hitting its network in the third quarter, a growth of 1.1 percent over the previous quarter and a surprising 11 percent increase over the past year.

The United States, which has the largest allocations of IPv4 addresses, saw the number of IPv4 addresses grow by 9.3 percent over the past year.

This growth and the slow, spotty uptake of IPv6 mask the fact that the pool of available IPv4 addresses continues to shrink. ARIN, the American Registry for Internet Numbers, is down to its last two/8 blocks of IPv4 addresses of 16.7 million each, making large pools of the addresses difficult to obtain. Inevitably, IPv6 will be growing.

Posted by William Jackson on Jan 31, 2014 at 11:14 AM0 comments


Congress

After Target cyberattack, Congress votes to do precisely nothing

The latest cybersecurity bill to be introduced in Congress took a small step forward last week. After the legislation passed out of a House subcommittee, its co-sponsors released a statement saying that, “the recent Target incident in which 110 million Americans’ personal information was compromised only underscores the very real and serious nature of the cyberthreat today.”

In response to the unprecedented attack, the National Cybersecurity and Critical Infrastructure Protection Act of 2013 (H.R. 3696) does precisely nothing. It is not just that the bill fails to do anything. Its purpose is actually to avoid doing anything and to codify the status quo; a policymaking status that current events have repeatedly shown to be inadequate.

Recognizing that our national security inevitably is bound up with the security of the nation’s privately owned critical infrastructure, the Homeland Security Department has for some years been tasked with providing voluntary technical and operational assistance to the private sector. DHS supports these firms in cooperation with the agencies that have regulatory authority over specific financial sectors, such as financial services and energy.

But DHS never has had authority to go beyond just offering assistance and advice on best practices.

This is the situation that would be formalized under H.R. 3696. The bill, according to House Homeland Security Committee Chairman Rep. Michael McCaul (R-Texas), who introduced it in December, “prohibits new regulatory authority at DHS and is budget neutral.” That is, the department gets no power to do anything and gets no money to do it. Instead, it codifies existing efforts such as the National Cybersecurity and Communications Integration Center, the National Infrastructure Protection Plan and the National Cybersecurity Incident Response Plan.

There is nothing wrong with these programs, as far as they go; which is not far enough. But the nation’s critical infrastructure is increasingly networked and accessible through the Internet, which exposes it to the full range of threats across the globe.

The emergence of complex, multistage exploits that quietly penetrate critical targets by leveraging vulnerabilities several links away from the target mean that it is difficult to be sure any system is effectively isolated. Because of this level of complexity and interconnectivity, it almost is impossible to find a system that might not be rated critical.

Given Congress’ record on passing cybersecurity legislation, the specific provisions of H.R. 3696 probably aren’t important. But it is disappointing to see that so many in Congress still refuse to acknowledge that the nation needs a strong baseline of protections for the systems on which its security and economy depend.

The belief has been that the private sector will set up effective cybersecurity on its own because it is in its interest to do so. But it has been shown over and over that this is not adequate.

Effective security cannot be legislated, and the last thing this nation needs is a technology prescription from politicians. But regulations with teeth that define required outcomes and responsibilities could go a long way toward ensuring that industry does what is needed to protect its own systems – and gets the assistance it needs from government.

Posted by William Jackson on Jan 24, 2014 at 7:57 AM0 comments


STEM

Drumbeat of data breaches sounds computer literacy alarms

According to the experts, there is a growing deficit of students and graduates with the skills needed to maintain and protect the nation’s IT systems. Jobs are waiting to be filled, but schools — particularly K-12 — are not providing the education needed to ready students for these jobs.

“We have a shortage of talent,” said Cisco’s chief security officer John Stewart. According to the company’s 2014 annual security report, there currently is a global shortage of 1 million security professionals at a time when the number and complexity of attacks against IT systems is growing. “Every enterprise is receiving more security alerts about services, software and hardware” that have to be evaluated, Stewart said, but there are not enough people to respond.

And government is not immune. In the past few months the Federal Election Commission was hacked, information on more than 1,500 persons was mistakenly mailed out by the VA Medical Center at Walla Walla, Wash., the Colorado governor’s office lost information on 18,800 state employees and Loudoun County Public Schools in Ashburn, Va., exposed student and staff data online.

The situation is only expected to get worse. At a recent hearing before the House Science, Space and Technology Subcommittee on Research and Technology, figures from the Bureau of Labor Statistics were cited predicting 1.4 million new computing jobs would be created in the next 10 years. Over the same time, however, the National Science Foundation predicted only 400,000 new computer science graduates would be available to fill them. 

Most of these jobs are not in tech companies, said Hadi Partovi, founder of Code.org, a nonprofit that promotes computer science education. An understanding of software and computers is required knowledge in the 21st century and needs to be taught in primary and secondary schools, he told lawmakers.

It is common knowledge that youngsters are tech savvy. But there is a difference in being able to use a device and understanding how it works. Real computer literacy, which involves some knowledge of programming and what is going on behind the interface, is something that must be taught. 

The need for more trained professionals has been recognized for some years now, and colleges and universities are stepping up to improve computer science education, including cybersecurity, in their graduate and undergraduate programs. But students are not graduating from high school with the skills needed to take advantage of these programs. 

The federal government spends about $3 billion a year to promote science, technology, engineering and math education, but only about 2 percent of that investment goes to computer science, said Partovi, and an alarming 90 percent of U.S. high schools have no formal computer science classes.

There is no question that kids like computers. Teaching them to understand devices beyond the touch screen should not be that much of a challenge. Doing so would benefit not only the students but the rest of our society as well.

Posted by William Jackson on Jan 17, 2014 at 8:42 AM0 comments