Monthly Archives: May 2015

Future of Xen Project – Video Spotlight with ARM’s Thomas Molgaard

ARM joined Xen Project two years ago as part of its drive into servers, networking and the emerging “Internet of Things” markets. In our latest “Future of Xen” video, Thomas Molgaard, Manager of Software Marketing – Systems & Software at ARM, talks about changes unfolding in enterprise and cloud computing that are creating new opportunities for his company and virtualization.

ARM designs the technology that is at the heart of advanced digital products, from wireless, networking and consumer entertainment solutions to imaging, automotive, security and storage devices. It offers electronics companies a comprehensive semiconductor IP portfolio, enhanced by the company’s broad partner community that increasingly embraces open source.

The company believes open source and its collaborative development model is keeping pace with transitions in the industry, helping to give companies more deployment options when it comes to cloud hosting, caching, scale-out storage and NoSQL and Hadoop analytics. ARM is hoping to offer even more variety to these application users. Early on the semiconductor design company recognized that Linux and Xen would play an important role in opening data centers up to enterprise-class 64-bit ARMv8 servers.  This recent eWeek article showcases a proof point from Cavium, one of the earliest vendors to launch ARM-based chips for servers, on display last week at OpenStack Summit in Vancouver.

Built from the ground up with open source best practices, Xen virtualization is increasingly deployed in applications targeted by ARM customers, including servers, networking infrastructure and embedded systems.  First to market with ARM support, Xen Project’s original ARM support focused on newer CPUs designed for servers. Taking direction from the community, ARM and Xen have expanded their scope to mobile, tablet, automotive, Internet of Things, midddlebox processing and other embedded applications.

In the video, Molgaard describes how Xen’s lean architecture is perfectly suited to ARM architecture-based solutions. Collaboration with the open source partners like Xen Project, Linaro and The Linux Foundation is extremely valuable as ARM makes further inroads in the data center and cloud infrastructure, he says.


Xen Project now in OpenStack Nova Hypervisor Driver Quality Group B

A few weeks ago, we introduced the Xen Project – OpenStack CI Loop, which is testing Nova commits against the Xen Project Hypervisor and Libvirt. Xen Project community is pleased to announce that we have moved from Quality Group C to B, as we’ve made significant progress in the last few weeks and the Xen Project CI loop is now voting on Nova commits.

This diagram shows the number of OpenStack Nova drivers for Hypervisors, which allow you to choose which Hypervisor(s) to use for your Nova Deployment. Note that these are classified into groups A, B and C. Xen Project is now in Quality Group B.

This diagram shows the number of OpenStack Nova drivers for Hypervisors, which allow you to choose which Hypervisor(s) to use for your Nova Deployment. Note that these are classified into groups A, B and C. Xen Project is now in Quality Group B.

Quality groups are defined as follows:

  • Group C: These drivers have minimal testing and may or may not work at any given time. Test coverage may include unit tests that gate commits. There is no public functional testing.
  • Group B: Test coverage includes unit tests that gate commits. Functional testing is provided by an external system (such as our CI loop) that do not gate commits, but advises patch authors and reviewers of results in OpenStack Gerrit.
  • Group A: Test coverage includes unit tests and functional testing that both gate commits.

What does this mean in practice?

The easiest way to understand what this means in practice, is to look at a real code review as shown in the figure below.

This diagram shows how the functional tests for the Xen Project initially failed, and passed after a new patch was uploaded.

This diagram shows how the functional tests for the Xen Project initially failed, and passed after a new patch was uploaded.

This code review shows that the OpenStack Jenkins instance (running on KVM) and the Xen Project CI Loop failed their respective tests initially, so a new patchset was uploaded and the tests succeeded afterwards.

This diagram shows the status after a new patchset was uploaded. Note that the review of the patchset is still pending manual review.

This diagram shows the status after a new patchset was uploaded. Note that the review of the patchset is still pending manual review.

Also see Merging.Repository Gating in the OpenStack documentation.

Relevant OpenStack Summit Sessions

There are a number of sessions at this week’s OpenStack Summit that are worth attending including:

Hands-on sessions to improve 3rd party CI infrastructure include:

Other third-party CI-related sessions include:

Relevant Regular Meetings

Note that there are also weekly Third Party CI Working Group meetings for all operators of 3rd party CI loops in #openstack-meeting-4 on Wednesdays at 1500/0400 UTC alternating organized by Kurt Taylor (krtaylor). Third party CI operators interested in enhancing documentation, reviewing patches for relevant work, and improving the consumability of infra CI components are encouraged to attend. See here for more information on the working group.

Hardening Hypervisors Against VENOM-Style Attacks

This is a guest blog post by Tamas K. Lengyel, a long-time open source enthusiast and Xen contributor. Tamas works as a Senior Security Researcher at Novetta, while finishing his PhD on the topic of malware analysis and virtualization security at the University of Connecticut.

The recent disclosure of the VENOM bug affecting major open-source hypervisors, such as KVM and Xen, has many reevaluating their risks when using cloud infrastructures. That is a very good thing indeed. Complex software systems are prone to bugs and errors.  Virtualization and the cloud have been erroneously considered by many to be a silver bullet against intrusions and malware. The fact is the cloud is anything but a safe place. There are inherent risks in the cloud and it’s very important to put those risks in their proper context.


VENOM is one of many vulnerabilities involving hypervisors (e.g. [Qemu], [ESX], [KVM], [Xen]). And, while Venom is indeed a serious bug and can result in a VM escape, which in turn can compromise all VMs on the host, it doesn’t have to be. In fact, we’ve known about VENOM-style attacks for a long time. Yet, there are easy-to-deploy counter-measures to mitigate the risk of such exploits natively available in both Xen and KVM (see RedHat’s blog post on the same topic).

What is the root cause of VENOM? Emulation

While modern systems come with a plethora of virtualization extensions, many components are still being emulated, usually devices such as your network card, graphics card and your hard drive. While Linux comes with paravirtual (virtualization-aware) drivers to create such devices, emulation is often the only solution to run operating systems that do not have kernel drivers. This has been traditionally the case with Windows (note that Citrix recently open-sourced its Windows PV drivers so this is no longer the case). This emulation layer has been implemented in QEMU, which caused VENOM and a handful of other VM-escape bugs in recent years. However, QEMU is not the only place where emulation is used within Xen. For a variety of interrupts and timers, such as RTC, PIT, HPET, PMTimer, PIC, IOAPIC and LAPIC, there is a baked-in emulator in Xen itself for performance and scalability reasons. As with QEMU, this emulator has been just as exploitable. This is simply because programming emulators properly is really complex and error-prone.

Available Mitigations

We’ve known how complicated emulation is for some time. In 2011, this Black Hat presentation on Virtunoid demonstrated a VM escape attack against KVM through QEMU. The author of Virtunoid even cautioned there would be many bugs of that nature found by researchers. Fast-forward four years and we now have VENOM.

The good news is it’s easy to mitigate all present and future QEMU bugs, which the recent Xen Security Advisory emphasized as well. Stubdomains can nip the whole class of vulnerabilities exposed by QEMU in the bud by moving QEMU into a de-privileged domain of its own. Instead of having QEMU run as root in dom0, a stubdomain has access only to the VM it is providing emulation for. Thus, an escape through QEMU will only land an attacker in a stubdomain, without access to critical resources. Furthermore, QEMU in a stubdomain runs on MiniOS, so an attacker would only have a very limited environment to run code in (as in return-to-libc/ROP-style), having exactly the same level of privilege as in the domain where the attack started. Nothing is to be gained for a lot of work, effectively making the system as secure as it would be if only PV drivers were used.

While it might sound complex, it’s actually quite simple to take advantage of this protection. Simply add the following line to your Xen domain configuration, and you gain immediate protection against VENOM and the whole class of vulnerabilities brought to you by QEMU:

device_model_stubdomain_override = 1

However, as with most security systems, it comes at a cost. Running stubdomains requires a bit of extra memory as compared to the plain QEMU process in dom0. This in turn can limit the number of VMs a cloud provider may be able to sell, so they have little incentive to enable this feature by default on their end. However, this protection works best if all VMs on the server run with stubdomains. Otherwise you may end up protecting your neighbors against an escape attack from your domain, while not being protected from an escape attack from theirs. It’s no surprise that some users would not want to pay for this extra protection, even if it was available. But for private clouds and exclusive servers there is no reason why it shouldn’t be your default setting.

Xen Project Hackathon 15 Event Report

After spending almost a week in Shanghai for the Xen Project Hackathon it is time to write up some notes.

More than 48 delegates from Alibaba, Citrix, Desay SV Automotive, GlobalLogic, Fujitsu, Huawei, Intel, Oracle, Suse and Visteon Electronics attended the event, which covered a wide range of topics.

I wanted to thank Susie Li, Hongbo Wang and Mei Yu from Intel for funding and organizing the event.

zizhu_intel Before Registration People Arriving Group Picture


Xen Project Hackathons started originally as pure hackathons, but have over time evolved to follow the Open Space Unconference format, which we tested in 2012 and fully embraced in 2013. It appears to be one of the best formats to foster discussion and problem solving for groups of up to 50 people.

Besides providing an opportunity to meet face-to-face and build bridges, our hackathons have been very successful in tackling difficult issues, which require plenty of interaction. These issues range from modifying our development process and solving architecture problems to conducting difficult design discussions, coordinating inter-dependencies and sharing experiences. Of course we also write code and sometimes conduct live code reviews in smaller groups alongside the discussion sessions.

00019 00028 20150428_161138 20150428_170321

Discussed Topics

At the event, we covered topics such as:

  • Cadence of maintenance releases
  • Numbering of Xen Project Releases
  • Xen 4.6 Release Planning
  • Testing and Testing Frameworks
  • Hot-patching in the Xen Project Hypervisor
  • Changes to the COLO architecture and interdependencies with Migration v2
  • Possible Future Improvements to Live Migration
  • Upstreaming of Intel GVT-g
  • Automotive, including lessons learned on implementing graphics virtualization using OpenGL 2.0 and a walk through of a mediated graphics virtualization solution for the Imagination PowerVR SGX544 GPU on Xen and ARM
  • Xen and OpenStack
  • Evolution of Virtual Machine Introspection (including HW assistance) in the Xen Hypervisor
  • Vendor Strategies For Upgrading Xen in their products (e.g. from Xen 4.1.5 to 4.5)
  • Effectiveness of New Xen Project Security Policy

As usual, we will post summaries (or patches/RFC’s) from these discussions on xen-devel@ – I will also post links to follow-up discussions on our wiki.

Future Xen Project Developer Events in Asia

We’ve learned that the term hackathon is misleading for this event and confuses some of our attendees. Our hackathons are really more of an Architecture Workshop and Design Summit. For this reason, we will probably rename the Hackathon: for a current proposal on the new name check out this and this e-mail thread.

As the event was very successful and we have a growing, active developer community in China, we are considering holding another similar event in 2017 or a Xen Project Developer Summit at LinuxCon Japan in 2017. Stay tuned for more details.

20150427_205419 20150427_205410 20150427_211809 20150429_174059