Securing the virtual world

As agencies move to virtualization, researchers target potential vulnerabilities

Next month at the annual Black Hat conference in Las Vegas, noted security researcher Joanna Rutkowska plans to demonstrate how a malicious hacker could take control of Xen virtualization software.

'With our presentations, we take the game to the next level by studying how to compromise the hypervisor and what we can do to prevent it,' Rutkowska said in an e-mail response to questions. If she makes good on her promise, it will be another chink in the armor of virtualization.

Increasingly in the past year, security issues have beset VMware and Xen, the two largest operating system virtualization applications. These programs are not less secure than other enterprise applications ' in fact, security researchers have applauded the code underlying the applications ' but there is an inevitable lag time between when a new application hits the enterprise and when it gets incorporated into the security profile of large organizations.

'We see a lot of organizations rolling [virtualization] out first and only later dovetailing it into the security planning,' said Chris Farrow, director of product strategy at Fortisphere, at the recent RSA Conference 2008 in San Francisco. 'The problems of the past come back to bite us in new and different ways.'

VMware and Citrix Systems, which offers a commercially supported version of the open-source Xen virtualization software, downplay the severity of the findings by Rutkowska and others, saying that they apply only to development code or peripheral products. Besides, company officials say, if there are holes in virtualization software, they would still be the most difficult ways to enter a network.

But researchers say the claims these companies make about the robustness of their software are greatly overstated and could lead users to false complacency.

In any case, administrators creating virtualized environments ' especially for security reasons ' should be aware of these issues and make use of tools already available to protect against them.

Take the red pill

Sci-fi movie lovers might remember the scene in 'The Matrix' in which Neo, played by Keanu Reeves, must choose between the red pill that provides the disturbing though ultimately illuminating truth of his surroundings and the blue pill that maintains the illusion he's living under.

The security research community has seized on this idea with regard to virtualization. Red Pill is shorthand for a user being aware that he or she is operating in a virtual environment. As part of the SANS Institute's Security 517 class, 'Cutting- Edge Hacking Techniques,' instructor John Strand outlined security researchers' thinking. When logging on to a computer, they look to see if they are in a virtualized environment, because if they are, they can look for tools installed by the virtualization software in that environment. The tools could have vulnerabilities that could be used to gain admittance to the host machine.

The goal for many researchers is to break out of the virtualized container, Strand said. No one has done it yet. But gathering vulnerabilities is the first step.

Slowly but surely researchers are finding program errors that can help them penetrate a host. In March, security analyst firm iDefense and VMware disclosed a technique called directory traversal that allows someone using VMware Tools to peek into the host computer via a File➔Open command. VMware issued a patch.

Xen hasn't been immune, either. In October, security research firm Secunia showed how a Xen command was not properly checking user input at one point, allowing users to input a malicious command in a string of text. And last month, another researcher found a buffer overflow error in the program's video frame buffer, again allowing for insertion of a malicious command.

In 2005, the Homeland Security Department's Homeland Security Advanced Research Projects Agency awarded security research firm Intelguardians a $1.2 million contract to investigate whether VMware, Xen and Microsoft virtualization products could be compromised. Intelguardians found that all three could. Only VMware's ESX Server has thus far been resistant to attack, said Ed Skoudis, Intelguardians' founder and senior security consultant.

Virtualization software companies have started looking at ways to allow users to access memory directly to speed response time. Most notably, device manufacturers are starting to add input/output memory management units (IOMMUs) to their products, said Nand Mulchandani, VMware's senior director of product management and marketing. IOMMUs allow virtualized environments to directly access a slice of device memory for their needs while keeping other parts of the memory secure from snooping. In addition, other software-only approaches mimic IOMMUs' mapping techniques.

Although all these techniques improved performance and provided adequate security, some setups did not protect virtual environments from misbehaving drivers, said Paul Willmann, at the annual Usenix user conference, where he presented a paper he co-authored on the performance and security trade-offs of IOMMUs.

In addition to vulnerabilities, each virtualization program also has unique operational characteristics, which malicious attackers could exploit. At its conference in Boston, Usenix offered a class on securing virtual environments, taught by Phil Cox, a principal consultant at SystemExperts. Cox pointed out some of the more worrisome characteristics of Xen and VMware.

For instance, VMware offers the ability to move a virtual machine from one physical server to another on the fly, a feature called VMotion. This could be handy for moving applications to less-busy machines or for moving work off machines that are starting to fail. However, Cox said, users should be aware that when the environment is moved, it is moved in plain text. It is not encrypted. Theoretically, a sniffer between the two machines could easily capture all the content on the virtual machine.

Another characteristic of VMware is that the VMware Infrastructure appears to use the Tomcat application server, which serves as the interface to the browser. As a result, it has all the standard settings used by Tomcat ' and could fall prey to all the same vulnerabilities. 'If I had to break ESX, I would go after Tomcat,' Cox said.

In April, the Defense Information Systems Agency published the 'ESX Server Security Technical Implementation Guide' and noted a number of other characteristics that could lead to security failures. For instance, the ESX Server only supports one-way Challenge-Handshake Authentication Protocol for iSCSI communication with hard drives ' and does not allow use of more robust alternatives such as Kerberos, IP Security or public-key authentication methods. If virtual switches are used, someone in a virtualized environment could view traffic traveling to an iSCSI device from other virtual machines on the same virtual localarea network.

Xen has its own quirky characteristics, Cox said. For instance, the Xen- Center management console has no independent log-in mechanism. Start the software and you will be greeted with 'Press to login' ' no user credentials needed. 'If you can get access to the server, you can access XenCenter,' Cox said.

How serious are these errors? It depends whom you ask. Cracking passwords, intercepting application programming interface calls, sneaking in through the storage systems and sniffing network traffic are all ways malicious attackers could gain a foothold into a system through virtualization software, Cox said.

However, Citrix chief security strategist Kurt Roemer downplayed the impact of Xen's security vulnerabilities, noting that those found so far have been only in versions of the software under development.

Both the Secunia finding and the more-recent frame buffer overflow did 'not affect any of the published Xen implementations,' either the free version at, nor Citrix's commercial version. They were found, and fixed, in the developmental open-source versions of the software, Roemer said. 'Published Xen is configured in a secure way.'

Mulchandani said all the vulnerabilities that have been found in VMware software were in the company's free and add-on products, such as VMware Workstation. He asserts that researchers have not found any faults with the company's core product, ESX Server.

He also questions the interest in breaking out of the virtual machine in the first place.

'What will you do when you break into the host in the first place? You want to attack other machines, right?' Mulchandani said. 'It's an absolutely convoluted way to do it ' break into the hypervisor to break into another machine when all the machines are on the [same] network. There are 150 ways to break through the machine on a network.'

A hypervisor is the underlying platform on which all virtual machines run.

'If your internal data-center network is open to snooping, or tapping, you have bigger problems,' Mulchandani said. 'You've got serious issues.'

Muscling up

For better or worse, the federal government is using virtualization. Thus, improving the security of such software 'is a good research area for the Defense Department to be in,' said John McDermott, a researcher at the Naval Research Laboratory.

McDermott spearheads a project dubbed Xenon to tighten up the code of Xen. He said Xen eventually will go through Common Criteria and other forms of advanced security testing. In 2006, VMware ESX Server Version 2.5 was certified as meeting Common Criteria Level 2, making it usable in trusted defense networks.

As anyone who has put an application through Common Criteria testing knows, the process is arduous, with reviewers carefully examining the code for any shortcomings ( McDermott's team does not look for bugs per se but instead looks for ways that code could be more clearly expressed.

NRL's work is one of a few projects under way to introduce Xen to high-assurance environments. George Coker of the National Security Agency's National Information Assurance Research Lab is spearheading a project called Xen Security Modules, which users could customize to fit their security needs.

'XSM provides hook points in the kernel for pre- and postchecks on whether an operation is allowed,' Roemer said.

Other research efforts include Security- Enhanced Xen, another NSA-related project; IBM's sHype; and projects at Intel.

VMware has been working to shore up security, too. Administrators can turn to many tools and guidelines to help secure their VMware installations. DISA's Security Technical Information Guide offers a comprehensive list of measures, and VMware also offers a hardening guide. In addition, security vendor Tripwire offers a free test suite for checking ESX Server settings for secure configuration. (See the GCN online Extra, 'Resources,' for a list of links to those documents.)

Vizioncore offers virtualization monitoring software for VMware called vCharter Pro and virtualization backup software called vRanger Pro. Both can be configured into the safeguards, said George Pradel, chief security strategist at Vizioncore.

Perhaps most importantly, administrators and managers need to recognize that they must think about the impact of virtualization software as they would any other piece of software.

'People think of virtualization as this very different architecture,' Mulchandani said. 'It isn't actually.'


  • Records management: Look beyond the NARA mandates

    Pandemic tests electronic records management

    Between the rush enable more virtual collaboration, stalled digitization of archived records and managing records that reside in datasets, records management executives are sorting through new challenges.

  • boy learning at home (Travelpixs/

    Tucson’s community wireless bridges the digital divide

    The city built cell sites at government-owned facilities such as fire departments and libraries that were already connected to Tucson’s existing fiber backbone.

Stay Connected