Tag Archives: security

Virtual Machine Introspection: A Security Innovation With New Commercial Applications

The article from Lars Kurth, the Xen Project chairperson, was first published on Linux.com.

A few weeks ago, Citrix and Bitdefender launched XenServer 7 and Bitdefender Hypervisor Introspection, which together compose the first commercial application of the Xen Project Hypervisor’s Virtual Machine Introspection (VMI) infrastructure. In this article, we will cover why this technology is revolutionary and how members of the Xen Project Community and open source projects that were early adopters of VMI (most notably LibVMI and DRAKVUF) collaborated to enable this technology.

Evolving Security Challenges in Virtual Environments

Today, malware executes in the same context and with the same privileges as anti-malware software. This is an increasing problem, too. The Walking Dead analogy I introduced in this Linux.com article is again helpful. Let’s see how traditional anti-malware software fits into the picture and whether our analogy applies to anti-malware software.

In the Walking Dead universe, Walkers have taken over the earth, feasting on the remaining humans. Walkers are active all the time, and attracted by sound, eventually forming a herd that may overrun your defences. They are strong, but are essentially dumb. As we explored in that Linux.com article, people make mistakes, so we can’t always keep Walkers out of our habitat.

For this analogy, let’s equate Walkers with malware. Let’s assume our virtualized host is a village, consisting of individual houses (VMs) while the Hypervisor and network provides the infrastructure (streets, fences, electricity, …) that bind the village together.

Enter the world of anti-malware software: assume the remaining humans have survived for a while and re-developed technology to identify Walkers fast, destroy them quickly and fix any damage caused. This is the equivalent of patrols, CCTV, alarmed doors/windows and other security equipment, troops to fight Walkers once discovered and a clean-up crew to fix any damage. Unfortunately, the reality of traditional malware security technology can only be deployed within individual houses (aka VMs) and not on the streets of our village.

To make matters worse, until recently malware was relatively dumb. However, this has changed dramatically in the last few years. Our Walkers have evolved into Wayward Pine’s Abbies, which are faster, stronger and more intelligent than Walkers. In other words, malware is now capable of evading or disabling our security mechanisms.

What we need is the equivalent of satellite surveillance to observe the entire village, and laser beams to remotely destroy attackers when they try and enter our houses. We can of course also use this newfound capability to quickly deploy ground troops and clean-up personnel as needed. In essence that is the promise that Virtual Machine Introspection gives us. It allows us to address security issues from outside the guest OS without relying on functionality that can be rendered unreliable from the ground. More on that topic later.

From VMI in Xen to the First Commercial Application: A Tale of Collaboration

The development of Virtual Machine Introspection and its applications show how the Xen Project community is bringing revolutionary technologies to market.

The development of Virtual Machine Introspection and its applications show how the Xen Project community is bringing revolutionary technologies to market.

The idea of Virtual Machine Introspection for the Xen Project Hypervisor hatched at Georgia Tech in 2007, building on research by Tal Garfinkel and Mendel Rosenblum in 2003. The technology was first incorporated into the Xen Project Hypervisor via the XenAccess and mem-events APIs in 2009. To some degree, this was a response to VMware’s VMsafe technology, which was introduced in 2008 and deprecated in 2012, as the technology had significant limitations at scale. VMSafe was replaced by vShield, which is an agent-based, hypervisor-facilitated, file-system anti-virus solution that is effectively a subset of VMsafe.

Within the Xen Project software however, Virtual Machine Introspection technology lived on due to strong research interests and specialist security applications where trading off performance against security was acceptable. This eventually led to the creation of LibVMI (2010), which made these APIs more accessible. This provided an abstraction that eventually allowed exposure of a subset of Xen’s VMI functionality to other open source virtualization technologies such as KVM and QEMU.

In May 2013, Intel launched its Haswell generation of CPUs, which is capable of maintaining up to 512 EPT pointers from the VMCS via the #VE and VMFUNC extensions. This proved to be a potential game-changer for VMI, enabling hypervisor controlled and hardware enforced strong isolation between VMs with lower than previous overheads, which led to a collaboration of security researchers and developers from Bitdefender, Cisco, Intel, Novetta, TU Munich and Zentific. From 2014 to 2015, the XenAccess and mem-events APIs have been re-architected into the Xen Project Hypervisor’s new VMI subsystem, alt2pm and other hardware capabilities have been added, as well as support for ARM CPUs and a baseline that was production ready has been released in Xen 4.6.

Citrix and Bitdefender collaborated to bring VMI technology to market: XenServer 7.0 introduced its Direct Inspect APIs built on the Xen Projects VMI interface. It securely exposes the introspection capabilities to security appliances, as implemented by Bitdefender HVI.

What Can Actually Be Done Today?

Coming back to our analogy: what we need is the equivalent of satellite surveillance to observe the entire village. Does VMI deliver? In theory, yes: VMI makes it possible to observe the state of any virtual machine (house and its surroundings in the village), including memory and CPU state and to receive events when the state of the virtual machine changes (aka if there is any movement). In practice, the performance overhead of doing this is far too high, despite using hardware capabilities.

In our imagined world that is overrun by Walkers and Abbies, this is equivalent to not having the manpower to monitor everything, which means we have to use our resources to focus on high value areas. In other words, we need to focus on the suspicious activity on system perimeters (the immediate area surrounding each of our houses).

This focus is executed by monitoring sensitive memory areas for suspicious activity. When malicious activity is detected, a solution can take corrective actions on the process state (block, kill) or VM state (pause, shutdown) while collecting and reporting forensic details directly from a running VM.

Think of a laser beam on our satellite that is activated whenever an Abbie or Walker approaches our house. In technical terms, the satellite and laser infrastructure maps to XenServer’s Direct Inspect API, while the software which controls and monitors our data maps onto Bitdefenders Hypervisor Introspection.

It is important to stress that monitoring and remedial action takes place from the outside, using the hypervisor to provide hardware-enforced isolation. This means that our attackers cannot disable surveillance nor laser beams.

Of course, no security solution is perfect. This monitoring software may not always detect all suspicious activity, if that activity does not impact VM memory. This does not diminish the role of file-system-based security; we must still be vigilant, and there is no perfect defense. In our village analogy, we could also be attacked through underground infrastructure such as tunnels and canalisation. In essence this means we have to use VMI together with traditional anti-malware software.

How does VMI compare to traditional hypervisor-facilitated anti-virus solutions such as vShield? In our analogy, these solutions require central management of all surveillance equipment that is installed in our houses (CCTV, alarmed doors/windows, …) while the monitoring of events is centralized very much like a security control centre in our village hall. Albeit such an approach significantly simplifies monitoring and managing of what goes on within virtual machines, it does not deliver the extra protection that introspection provides.

You can find more information (including some demos) about VMI, XenServer Direct Inspect API and BitDefender Hypervisor Introspection here:

Xen Project Virtual Machine Introspection

Conclusion

The development of VMI and its first open source and commercial applications show how the Xen Project community is innovating in novel ways, and is capable of bringing revolutionary technologies to market. The freedom to see the code, to learn from it, to ask questions and offer improvements has enabled security researchers and vendors such as Citrix and Bitdefender to bring new solutions to market.

It is also worth pointing out that hardware-enabled security technology is moving very fast: only a subset of Intel’s #VE and VMFUNC extensions are currently being deployed in VMI. Making use of more hardware extensions carries the promise of combining the protection of out-of-guest tools with the performance of in-guest tools.

What is even more encouraging is that other vendors such as A1Logic, Star Lab and Zentific are working on new Xen Project-based security solutions. In addition, the security focused, Xen-based OpenXT project has started to work more closely with the Xen Project community, which promises further security innovation.

A few of these topics will be discussed in more detail during Xen Project Developer Summit happening in Toronto, CA from August 25 – 26, 2016. You learn more about the event here.

Q&A: Xen Project Release Strengthens Security and Pushes New Use Cases

The following Q&A with Lars Kurth, the Xen Project chairperson, was first published on Linux.com.

Xen Project technology supports more than 10 million users and is a staple in some of the largest clouds in production today, including Amazon Web Service, Tencent, and Alibaba’s Aliyun. Recently, the project announced the arrival of Xen Project Hypervisor 4.7. This new release focuses on improving code quality, security hardening and features, and support for the latest hardware. It is also the first release of the project’s fixed-term June – December release cycles. The fixed-term release cycles provide more predictability making it easier for consumers of Xen to plan ahead.

We recently sat down with the Xen Project chairperson, Lars Kurth, to talk about some of the key features of the release and the future of Xen Project technology. Lars will be discussing this topic and more during Xen Project’s Developer Summit in Toronto, CA from August 25-26 — the conference is directly after LinuxCon North America.

Q: What was the focus on this release?

Lars Kurth: There were five areas that we focused on for this release (full details are in our blog). In summary, we focused on security features, migration support, performance and workloads, support for new hardware features, and drivers and devices (Linux, FreeBSD and other).

Security is consistently something that we focus on in all of our releases. There are a lot of people that rely on Xen Project technology and security is our top concern in any release as well as how we organize our process around security disclosures.

Q: What was the biggest feature coming out of this release?

Lars: The biggest feature for us is live patching, which is a technology that enables re-boot free deployment for security patches to minimize disruption and downtime during security upgrades for cloud admins. It essentially eliminates all cloud reboots, making cloud providers and their users much more safe. It also eliminates a lot of headaches for system and DevOps admins of the world.

Q: Xen is often associated with the cloud, but are there additional use cases that you see growing around this technology, if so why?

Lars: We are seeing a lot of growth in terms of contributions, as well as many different use cases emerging, including automotive, aviation, embedded scenarios, security, and also IoT. In addition, we continue to grow within the public cloud sector and traditional server virtualization.

On the security front, for example, a number of vendors such as A1Logic, Bitdefender, Star Lab and Zentific have released or are working on new Xen Project-based security solutions. In addition, the security focused and Xen-based OpenXT project has started to work more closely with the Xen Project community.

Long-time contributors to the Xen Project, such as DornerWorks – a premier provider of electronic engineering services for the aerospace, medical, automotive, and industrial markets – have expanded their scope and are now providing support for the Xen Xilinx Zynq Distribution targeting embedded use-cases. We have also seen an increasing number of POCs and demos of automotive solutions, which include Xen as a virtualization solution.

Growth in these sectors is largely due to the Xen Project’s flexibility, extensibility, customisability and a clear lead when it comes to security-related technologies. Over the last year, we have also seen contributions increase from developers with strong security and embedded backgrounds. In fact, this totaled nearly 17 percent of the overall contributions in this release cycle, up from 9 percent in the previous release.

Q: How did you address these uses cases in this latest release?

Lars: We introduced the ability to remove core Xen Project Hypervisor features at compile via KCONFIG. This creates a more lightweight hypervisor and eliminates extra attack surfaces that are beneficial in security-first environments and microservice architectures. Users will still be able to get the core hypervisor functions, but they won’t receive all the drivers, schedulers, components or features that might not fit their use case.

Essentially it gives people an “a la carte” feature set. They can decide what they need for compliance, safety or performance reasons.

Q: Were there any new contributors for this release that surprised you?

Lars: We had three new companies contributing to the project: Star Lab, Bosch and Netflix. I met engineers from Star Lab for the first time at the 2015 Developer Summit less than a year ago, and helped introduce them to the Project’s culture. In that short period of time, Doug Goldstein from Star Lab has moved into the top five contributors and top 10 code reviewers for the Project.

I was surprised about Netflix’s contributions; I didn’t even know the company used Xen. Netflix improved and secured the VPMU feature, which is incredibly useful for system tuning and performance monitoring. Bosch Car Multimedia GmbH added some new ARM functionality. In addition, we have seen quite a bit of Xen related development in upstream and downstream projects such as Linux, FreeBSD, NetBSD, OpenBSD, QEMU and Libvirt.

Q: What’s next for Xen Project? Where do you think the technology is heading in the future and why?

Lars: In the last three releases, we introduced several major new features such as PVH, COLO, new schedulers, VMI, Live Patching, Graphics Virtualization, etc. and significant re-work of existing features such as Migration and the Xen Security Modules (XSM). Looking at trends within the community, I expect that stepwise evolution of large new features to continue.

Some new capabilities, such as restartable Dom0’s, and additional techniques to provide more isolation and security, are also likely to appear. In addition, it looks likely that we will see some GPU virtualization capabilities for GPUs that target the ARM ecosystem, although it is not yet clear whether these will be available as open source. I also expect that both Intel and ARM hardware features will be closely tracked.

Some areas, such as new schedulers, XSM, PVH and Live Patching, will see significant efforts to harden and improve existing functionality. The goal is to ensure their swift adoption in commercial products and Linux and BSD distributions. Some features, which are not enabled by default are likely to become part of the Xen Project Hypervisor’s default configuration.

Intel hosts OpenXT Summit on Xen Project based Client Virtualization, June 7-8 in Fairfax, VA, USA

This is a guest blog post by Rich Persaud, former member of the Citrix XenServer and XenClient engineering and business teams. He is currently a consultant to BAE Systems, working on the OpenXT project, which stands on the shoulders of the Xen Project, OpenEmbedded Linux and XenClient XT.

While the Xen Project is well known for servers and hosted infrastructure, Type-1 hypervisors have been used in client endpoints and network appliances, improving security and remote manageability. Virtualization-based security in Qubes and Windows 10 is also educating system administrators about hardware security (IOMMU and TPM) and application trust models.

Released as open-source software in 2014, OpenXT is a development toolkit for hardware-assisted security research and appliance integration. It includes hardened Linux VMs that can be configured as a user-facing software appliance for client devices, with virtualization of storage, network, input, sound, display and USB devices. Hardware targets include laptops, desktops and workstations.

OpenXT stands on the shoulders of the Xen Project, OpenEmbedded Linux and XenClient XT. It is optimized for hardware-assisted virtualization with an IOMMU and a TPM. It configures Xen network driver domains, Linux stub domains, Xen Security Modules, Intel TXT, SE Linux, GPU passthrough and VPNs. Guest operating systems include Windows, Linux and FreeBSD. VM storage options include encrypted VHD files with boot-time measurement and non-persistence.

The picture below shows a typical OpenXT software stack, including Xen, Linux and other components.

The picture above shows one of many configurations of the OpenXT software stack, including Xen, Linux and other components.

OpenXT enables loose coupling of open-source and proprietary software components, verifiable measurements of hardware and software, and verified launch of derivative products. It has been used to develop locally/centrally managed software appliances that isolate high-risk workloads, networks and devices.

The inaugural OpenXT Summit brings together developers and ecosystem participants for a 2-day conference in Fairfax, VA, USA on June 7-8, 2016. The event is hosted by Intel Corporation. The audience for this event includes kernel and application developers, hardware designers, system integrators and security architects.

The 2016 OpenXT Summit will chart the evolution of OpenXT from cross-domain endpoint virtualization to an extensible systems innovation platform, enabling derivative products to make security assurances for diverse hardware, markets and use cases.

The Summit includes one day of presentations, a networking reception and one day of moderated technical discussions. Presentation topics will include OpenXT architecture, TPM 2.0, Intel SGX, Xen security, measured launch, graphics virtualization and NSA research on virtualization and trusted computing.

For more information, please see the event website at http://openxt.org/summit.

For presentations and papers related to OpenXT, please see http://openxt.org/history.

Stealthy monitoring with Xen altp2m

One of the core features that differentiates Xen from other open-source hypervisors is its native support for stealthy and secure monitoring of guest internals (aka. virtual machine introspection [1]). In Xen 4.6 which was was released last autumn several new features have been introduced that make this subsystem better; a cleaned-up, optimized API and ARM support being just some of the biggest items on this list. As part of this release of Xen, a new and unique feature was also successfully added by a team from Intel that make stealthy monitoring even better on Xen: altp2m. In this blog entry we will take a look at what it’s all about.

p2mIn Xen’s terminology, p2m stands for the memory management layer that handles the translation from guest physical memory to machine physical. This translation is critical for safely partitioning the real memory of the machine between Xen and the various VMs running as to ensure a VM can’t access the memory of another without permission. There are several implementations of this mechanism, including one with hardware support via Intel Extended Page Tables (EPT) available to HVM guests and PVH . In Xen’s terminology, this is called Hardware Assisted Paging (hap). In this implementation the hypervisor maintains a second pagetable, similar to the one in 64-bit operating systems use, dedicated to running the p2m translation. All open-source hypervisors that use this hardware assisted paging method use a single EPT per virtual machine to handle this translation, as most of the time the memory of the guest is assigned at VM creation and doesn’t change much afterwards.

altp2mXen altp2m is the first implementation which changes this setup by allowing Xen to create more then one EPT for each guest. Interestingly, the Intel hardware has been capable of maintaining up to 512 EPT pointers from the VMCS since the Haswell generation of CPUs. However, no hypervisor made use of this capability until now. This changed in Xen 4.6, where we can now create of up to 10 EPTs per guest. The primary reason for this feature is to use it with the #VE and VMFUNC extensions.

It can also be used by external monitoring applications via the Xen vm_event system.

Why alt2pm is a game-changer

Alt2pm is a game-changer for applications performing purely external monitoring is because it simplifies the monitoring process of multi-vCPU guests. The EPT layer has been successfully used in stealthy monitoring applications to track the memory accesses made by the VM from a safe vantage point by restricting the type of access the VM may perform on various memory pages. Since EPT permission violations trap into the hypervisor, the VM would receive no indication that anything out of the ordinary has happened. While the method allowed for stealthy tracing of R/W/X memory accesses of the guest, the memory permission needs to be relaxed in order to allow the guest to continue execution. When a single EPT is shared across multiple running vCPUs, relaxing the permissions to allow one vCPU to continue may inadvertently allow another one to perform the memory access we would otherwise want to track. While under normal circumstances such race-condition may rarely occur, malicious code could easily use this to hide some of its actions from a monitoring application.

Solutions to this problem exist already. For example we can pause all vCPUs while the one violating the access is single-stepped. This approach however introduces heavy overhead just to avoid a race-condition that may rarely occur in practice. Alternatively, one could emulate the instruction that was violating the EPT permission without relaxing the EPT access permissions, as Xen’s built-in emulator doesn’t use EPT to access the guest memory. This solution, while supported in Xen, is not particularly ideal either as Xen’s emulator is incomplete and is known to have issues that can lead to guest instability [2]. Furthermore, over the years emulation has been a hotbed of various security issues in many hypervisors (including Xen [3]), thus building security tools based on emulation is simply asking for trouble. It can be handy but should be used only when no other option is available.

Xen’s altp2m system changes this problem quite significantly. By having multiple EPTs we can have differing access permissions defined in each table, which can be easily swapped around by changing the active EPT index in the VMCS. When the guest makes a memory access that is monitored, instead of having to relax the access permission, Xen can simply switch to an EPT (called a view) that allows the operation to continue. Afterwards the permissive view can be switched back to the restricted view to continue monitoring. Since each vCPU has its own VMCS where this switching is performed, this monitoring can be performed specific to each vCPU, without having to pause any of the other ones, or having to emulate the access. All without the guest noticing any of this switching at all. A truly simple and elegant solution.

Other introspection methods for stealthy monitoring

EPT based monitoring is not the only introspecting technique used for stealthy monitoring. For example, the Xen based DRAKVUF Dynamic Malware Analysis [4] uses it in combination with an additional technique to maximum effect. The main motivation for that is because EPT based monitoring is known to introduce significant overhead, even with altp2m: the granularity of the monitoring is that of a memory page (4KB). For example, if the monitoring application is really just interested in when a function-entry point is called, EPT based monitoring creates a lot of “false” events when that page is accessed for the rest of the function’s code.

This can be avoided by enabling the trapping of debug instructions into the hypervisor, a built-in feature of Intel CPUs that Xen exposes to third-party applications. This method is used in DRAKVUF, which writes breakpoint instructions into the guests’ memory at code-locations of interest. Since we will only get an event for precisely the code-location we are interested in this method effectively reduces the overhead. However, the trade-off is that unlike EPT permissions the breakpoints are now visible to the guest. Thus, to hide the presence of the breakpoints from the guest, these pages need to get further protected by restricting the pages to be execute-only in the EPT. This allows DRAKVUF to remove the breakpoints before in-guest code-integrity checking mechanisms (like Windows Patchguard) can access the page. While with altp2m the EPT permissions can be safely used with multi-vCPU systems, using breakpoints similarly presents a race-condition: the breakpoint hit by one vCPU has to be removed to allow the guest to execute the instruction that was originally overwritten, potentially allowing another vCPU to do so as well without notice.

altp2m-shadowFortunately, altp2m has another neat feature that can be used to solve this problem. Beside allowing for changing the memory permissions in the different altp2m views, it also allows to change the mapping itself! The same guest physical memory can be setup to be backed by different pages in the different views. With this feature we can really think of guest physical memory as “virtual”: where it is mapped really depends on which view the vCPU is running on. Using this feature allows us to hide the presence of the breakpoints in a brand new way. To do this, first we create a complete shadow copy of the memory page where a breakpoint is going to be written and only write the breakpoint into this shadow copy. Now, using altp2m, we setup a view where the guest physical memory of the page gets mapped to our shadow copy. The guest continues to access its physical memory as before, but underneath it is now using the trapped shadow copy. When the breakpoint is hit, or if something is trying to scan the code, we simply switch the view to the unaltered view for the duration of a single-step, then switch back to the trapped view. This allows us to hide the presence of the breakpoints specific to each vCPU! All without having pause any of the other vCPUs or having to emulate. The first open-source implementation of this tracing has been already merged into the DRAKVUF Malware Analysis System and is available as a reference implementation for those interested in more details.

Conclusion

As we can see, Xen continues to be on the forefront of advancing the development of virtualization based security application and allowing third-party tools to create some very exotic setups. This flexibility is what’s so great about Xen and why it will continue to be a trend-setter for the foreseeable future

References

[1] Virtual Machine Introspection
[2] xen-devel@: Failed vm entry with heavy use of emulator
[3] Hardening Hypervisors Against VENOM-Style Attacks
[4] DRAKVUF Malware Analysis System (drakvuf.com)
[5] Stealthy, Hypervisor-based Malware Analysis (Presentation)

Security vs Features

We’ve just released a rather interesting batch of Xen security advisories. This has given rise in some quarters to grumbling around Xen not taking security seriously.

I have a longstanding interest in computer security. Nowadays I am a member of the Xen Project Security Team (the team behind security@xenproject, which drafts the advisories and coordinates the response). But I’m going to put forward my personal opinions.

Of course Invisible Things are completely right that security isn’t taken seriously enough. The general state of computer security in almost all systems is very poor. The reason for this is quite simple: we all put up with it. We, collectively, choose convenience and functionality: both when we decide which software to run for ourselves, and when we decide what contributions to make to the projects we care about. For almost all software there is much stronger pressure (from all sides) to add features, than to improve security.

That’s not to say that many of us working on Xen aren’t working to improve matters. The first part of improving anything is to know what the real situation is. Unlike almost all corporations, and even most Free Software projects, the Xen Project properly discloses, via an advisory, every vulnerability discovered in supported configurations. We also often publish advisories about vulnerabilities in other relevant projects, such as Linux and QEMU.

Security bugs are bugs, and over the last few years the Xen Project’s code review process has become a lot more rigorous. As a result, the quality of code being newly introduced into Xen has improved a lot.

For researchers developing new analysis techniques, Xen is a prime target. A significant proportion of the reports to security@xenproject are the result of applying new scanning techniques to our codebase. So our existing code is being audited, with a focus on the areas and techniques likely to discover the most troublesome bugs.

As I say, the Xen Project is very transparent about disclosing security issues; much more so than other projects. This difference in approach to disclosure makes it difficult to compare the security bug density of competing systems. When I worked for a security hardware vendor I was constantly under pressure to explain why we needed to do a formal advisory for our bugs. That is what security-conscious users expect, but our competitors’ salesfolk would point to our advisories and say that our products were full of bugs. Their products had no publicly disclosed security bugs, so they would tell naive customers that their products had no bugs.

I do think Xen probably has fewer critical security bugs than other hypervisors (whether Free or proprietary). It’s the best available platform for building high security systems. The Xen Project’s transparency is very good for Xen’s users. But that doesn’t mean Xen is good enough.

Ultimately, of course, a Free Software project like Xen is what the whole community makes it. In the project as a whole we get a lot more submissions of new functionality than we get submissions aimed at improving the security.

So personally, I very much welcome the contributions made by security-focused contributors – even if that includes criticism.

Hardening Hypervisors Against VENOM-Style Attacks

This is a guest blog post by Tamas K. Lengyel, a long-time open source enthusiast and Xen contributor. Tamas works as a Senior Security Researcher at Novetta, while finishing his PhD on the topic of malware analysis and virtualization security at the University of Connecticut.

The recent disclosure of the VENOM bug affecting major open-source hypervisors, such as KVM and Xen, has many reevaluating their risks when using cloud infrastructures. That is a very good thing indeed. Complex software systems are prone to bugs and errors.  Virtualization and the cloud have been erroneously considered by many to be a silver bullet against intrusions and malware. The fact is the cloud is anything but a safe place. There are inherent risks in the cloud and it’s very important to put those risks in their proper context.

venom

VENOM is one of many vulnerabilities involving hypervisors (e.g. [Qemu], [ESX], [KVM], [Xen]). And, while Venom is indeed a serious bug and can result in a VM escape, which in turn can compromise all VMs on the host, it doesn’t have to be. In fact, we’ve known about VENOM-style attacks for a long time. Yet, there are easy-to-deploy counter-measures to mitigate the risk of such exploits natively available in both Xen and KVM (see RedHat’s blog post on the same topic).

What is the root cause of VENOM? Emulation

While modern systems come with a plethora of virtualization extensions, many components are still being emulated, usually devices such as your network card, graphics card and your hard drive. While Linux comes with paravirtual (virtualization-aware) drivers to create such devices, emulation is often the only solution to run operating systems that do not have kernel drivers. This has been traditionally the case with Windows (note that Citrix recently open-sourced its Windows PV drivers so this is no longer the case). This emulation layer has been implemented in QEMU, which caused VENOM and a handful of other VM-escape bugs in recent years. However, QEMU is not the only place where emulation is used within Xen. For a variety of interrupts and timers, such as RTC, PIT, HPET, PMTimer, PIC, IOAPIC and LAPIC, there is a baked-in emulator in Xen itself for performance and scalability reasons. As with QEMU, this emulator has been just as exploitable. This is simply because programming emulators properly is really complex and error-prone.

Available Mitigations

We’ve known how complicated emulation is for some time. In 2011, this Black Hat presentation on Virtunoid demonstrated a VM escape attack against KVM through QEMU. The author of Virtunoid even cautioned there would be many bugs of that nature found by researchers. Fast-forward four years and we now have VENOM.

The good news is it’s easy to mitigate all present and future QEMU bugs, which the recent Xen Security Advisory emphasized as well. Stubdomains can nip the whole class of vulnerabilities exposed by QEMU in the bud by moving QEMU into a de-privileged domain of its own. Instead of having QEMU run as root in dom0, a stubdomain has access only to the VM it is providing emulation for. Thus, an escape through QEMU will only land an attacker in a stubdomain, without access to critical resources. Furthermore, QEMU in a stubdomain runs on MiniOS, so an attacker would only have a very limited environment to run code in (as in return-to-libc/ROP-style), having exactly the same level of privilege as in the domain where the attack started. Nothing is to be gained for a lot of work, effectively making the system as secure as it would be if only PV drivers were used.

While it might sound complex, it’s actually quite simple to take advantage of this protection. Simply add the following line to your Xen domain configuration, and you gain immediate protection against VENOM and the whole class of vulnerabilities brought to you by QEMU:

device_model_stubdomain_override = 1

However, as with most security systems, it comes at a cost. Running stubdomains requires a bit of extra memory as compared to the plain QEMU process in dom0. This in turn can limit the number of VMs a cloud provider may be able to sell, so they have little incentive to enable this feature by default on their end. However, this protection works best if all VMs on the server run with stubdomains. Otherwise you may end up protecting your neighbors against an escape attack from your domain, while not being protected from an escape attack from theirs. It’s no surprise that some users would not want to pay for this extra protection, even if it was available. But for private clouds and exclusive servers there is no reason why it shouldn’t be your default setting.