Intel hosts OpenXT Summit on Xen Project based Client Virtualization, June 7-8 in Fairfax, VA, USA

This is a guest blog post by Rich Persaud, former member of the Citrix XenServer and XenClient engineering and business teams. He is currently a consultant to BAE Systems, working on the OpenXT project, which stands on the shoulders of the Xen Project, OpenEmbedded Linux and XenClient XT.

While the Xen Project is well known for servers and hosted infrastructure, Type-1 hypervisors have been used in client endpoints and network appliances, improving security and remote manageability. Virtualization-based security in Qubes and Windows 10 is also educating system administrators about hardware security (IOMMU and TPM) and application trust models.

Released as open-source software in 2014, OpenXT is a development toolkit for hardware-assisted security research and appliance integration. It includes hardened Linux VMs that can be configured as a user-facing software appliance for client devices, with virtualization of storage, network, input, sound, display and USB devices. Hardware targets include laptops, desktops and workstations.

OpenXT stands on the shoulders of the Xen Project, OpenEmbedded Linux and XenClient XT. It is optimized for hardware-assisted virtualization with an IOMMU and a TPM. It configures Xen network driver domains, Linux stub domains, Xen Security Modules, Intel TXT, SE Linux, GPU passthrough and VPNs. Guest operating systems include Windows, Linux and FreeBSD. VM storage options include encrypted VHD files with boot-time measurement and non-persistence.

The picture below shows a typical OpenXT software stack, including Xen, Linux and other components.

The picture above shows one of many configurations of the OpenXT software stack, including Xen, Linux and other components.

OpenXT enables loose coupling of open-source and proprietary software components, verifiable measurements of hardware and software, and verified launch of derivative products. It has been used to develop locally/centrally managed software appliances that isolate high-risk workloads, networks and devices.

The inaugural OpenXT Summit brings together developers and ecosystem participants for a 2-day conference in Fairfax, VA, USA on June 7-8, 2016. The event is hosted by Intel Corporation. The audience for this event includes kernel and application developers, hardware designers, system integrators and security architects.

The 2016 OpenXT Summit will chart the evolution of OpenXT from cross-domain endpoint virtualization to an extensible systems innovation platform, enabling derivative products to make security assurances for diverse hardware, markets and use cases.

The Summit includes one day of presentations, a networking reception and one day of moderated technical discussions. Presentation topics will include OpenXT architecture, TPM 2.0, Intel SGX, Xen security, measured launch, graphics virtualization and NSA research on virtualization and trusted computing.

For more information, please see the event website at http://openxt.org/summit.

For presentations and papers related to OpenXT, please see http://openxt.org/history.

Announcing Xen Project 4.7 RC and Test Day Schedule

Yesterday we created Xen 4.7 RC2 and will release a new release candidate every Wednesday, until we declare a release candidate as the final candidate and cut the Xen 4.7 release. We will also hold a Test Day every Friday for the release candidate that was released the Wednesday prior to the Test Day. This means we will have Test Days on May 13th, 20th, 27th and June 3rd. Your testing is still valuable on other days, so please feel free to send Test Reports as outlined below at any time.

Getting, Building and Installing a Release Candidate

Release candidates are available from our git repository at

git://xenbits.xen.org/xen.git (tag 4.7.0-<rc>)

where <rc> is rc1, rc2, rc3, etc. and as tarball from

http://bits.xensource.com/oss-xen/release/4.7.0-<rc>/xen-4.7.0-<rc>.tar.gz
http://bits.xensource.com/oss-xen/release/4.7.0-<rc>/xen-4.7.0-<rc>.tar.gz.sig

Detailed build and Install instructions can be found on the Test Day Wiki.

Testing new Features, Test and Bug Reports

You can find Test Instructions for new features on our Test Day Wiki and instructions for general tests on Testing Xen. The following pages provide information on how to report successful tests and how to report bugs and issues.

Happy Testing!

Please Welcome new Members of the Xen Project Hypervisor Leadership Team

Evolution of Hypervisor Git Commits within the project. Note that in in the same time period, the number of individuals and organisations contributing to the project has nearly doubled.

Evolution of Hypervisor Git Commits within the project. Note that in parallel the number of individuals and organisations contributing to the project has nearly doubled.

The Xen Project has experienced incredible growth in our community (see diagram on the right) and simultaneously the Xen Project advisory board has funded a lot of great projects that help support the larger Xen Project ecosystem, for example MirageOS, a library operating system that constructs unikernels for secure, high-performance network applications across a variety of cloud computing and mobile platforms. These projects are extremely important to the expansion and betterment of virtualization and cloud computing infrastructure, but also demand more work to be done by committers and maintainers.

We understood that there was plenty of leadership among the community, but didn’t know the best way to promote contributors to maintainers and committer roles for leadership to the Hypervisor and its interface with Linux.

We decided to introduced a new convention, by which we actively reminded community members to nominate or self-nominate themselves for leadership roles. Often times, active developers simply worked on the project, but did not consider to nominate newcomers (or themselves) for these leadership roles within the project.

We also felt that running the entire nomination process in public, which may include public feedback on a nominee, could discourage people from recommending themselves or recommending others. So we decided to follow the approach that Debian uses for Technical Committee seats, where candidates are nominated in private. A group of the senior leadership for the Xen Project would then review the submissions and provide feedback and acceptance, making the overall process less intimidating.

As for committers and maintainers, we promoted several key Xen Project contributors to these distinguished roles based on previous experience and similar work they had already been performing, but didn’t have the authorship to own. This very important as it will ensure the group of maintainers that we currently have will have the support they need to accommodate for the Xen Project hypervisor.

Through taking the approach of consistent promotion and private forums, we have found some incredible new members to the Xen Project Hypervisor Leadership team that we want to introduce to you.

Committers
The following people have been elected to be new Committers to the project, they will be joining long-time committers Ian Campbell, Ian Jackson, Jan Beulich and Konrad Rzeszutek Wilk:

Andrew Cooper has been working on the Hypervisor since 2011 and has added a number of major new features such as Migration v2, significant change to trap handling, improvements to cpuid handling for guests and many more.

George Dunlap has been working on the Hypervisor since 2005 and was heavily involved in making the tracing system useable for performance analysis, optimising the shadow code, wrote the credit2 scheduler and developed many other significant features and improvements in the hypervisor. In addition, he was our first Release Manager and is leading the CentOS Virtualisation SIG within CentOS.

Stefano Stabellini has been working on the Hypervisor and the Linux Kernel since 2007 and was instrumental in bringing ARM support to the Xen Hypervisor. He has also been leading many other activities within the project, such as the creation of libxenlight, adding support upstream QEMU to Xen, Xen OpenStack integration and Raisin.

Wei Liu started to work on the Xen Project as a GSoC student in 2011 (working in virtio support). He has been working on libxl support, event channel scalability, MiniOS and many other major Xen features. In addition, he has been the Xen Project Release Manager since Xen Project 4.6 release.

Andrew and Wei celebrated their appointment at the Xen Project hackathon last month by submitting and ACKing a piece of code while on a punt on the river Cam in Cambridge, UK.

Security Team
In addition, Andrew Cooper and George Dunlap are now also members of the Xen Project Security Team, alongside Ian Jackson, Jan Beulich, Konrad Rzeszutek Wilk and Tim Deegan.

Maintainers
The following people were also recently added as MAINTAINERS of the project: Doug Goldstein (KConfig, Travis CI), Julien Grall (ARM support, device tree, …), Meng Xu (RTDS Scheduler) and Paul Durrant (x86 I/O emulation, x86 viridian enlightenments, …). In addition, we clarified some ambiguities around the maintainer role.

Linux Kernel Maintainers
Jürgen Gross who has been a Linux kernel and Xen developer since 2004, but has significantly increased his engagement within the community in the last two years, is now Linux Kernel maintainer for the Xen Hypervisor Interface alongside Boris Ostrovsky and David Vrabel. Other maintainers of Xen specific components in the Linux Kernel are Stefano Stabellini, Wei Lui, and Konrad Rzeszutek Wilk.

A couple of months ago two of our committers, Keir Fraser and Tim Deegan, formally stepped down in their roles as committers from the Hypervisor team. We want to thank Keir and Tim for the vast contributions to the project. We look forward to seeing what they work on next and, again, thank them for the success that they brought to the open source Xen hypervisor.

Xen Project Hackathon 16 : Event Report

We just wrapped another successful Xen Project Hackathon, which is an annual event, hosted by Xen Project member companies, typically at their corporate offices. This year’s event was hosted by ARM at their Cambridge HQ. 42 delegates descended on Cambridge from Aporeto, ARM, Assured Information Security, Automotive Electrical Systems, BAE Systems, Bromium, Citrix, GlobalLogic, OnApp, Onets, Oracle, StarLab, SUSE and Vates to attend. A big thank you (!) to ARM and in particular to Thomas Molgaard for organising the event and the social activities afterwards.

Here are a few images that helped capture the event:

Taking a breather and photo opp outside of ARM headquarters in Cambridge

Taking a breather and photo opp outside of ARM headquarters in Cambridge

Working on solving the mysteries of the world.

Working on solving the mysteries of the world

Continuing to work hard on solving the mysteries of the world

Continuing to work hard on solving the mysteries of the world

Xen Project Hackathons have evolved in format into a series of structured problem solving sessions that scale up to 50 people. We combine this with a more traditional hackathon approach where programmers (and others involved in software development) collaborate intensively on software projects.

This year’s event was particularly productive because all our core developers and the project’s leadership were present. We focused on a lot of topics, but two of our main themes this year evolved around security and community development. We’ll cover these topics in more detail and how they fit within our next release 4.7 and development going forward, but below is a little taste of some of the other themes of this year’s Hackathon sessions:

  • Security improvements: A trimmed down QEMU to reduce attack surface, de-privileging QEMU and the x86 emulator to reduce the impact of security vulnerabilities in those components, XSplice, KConfig support which allows to remove parts of Xen at compile time, run-time disablement of Xen features to reduce the attack surface, vulnerabilities, disaggregation and enabling XSM (Xen’s equivalent of the Linux Security Modules which are also known as SELinux) by default.
  • Security features: We had two sessions on the future of XSplice (first version to be released in Xen 4.7), which allows users of Xen to apply security fixes on a running Xen instance (aka no need to reboot).
  • Robustness: A session on restartable Dom0 and driver domains, which again will significantly reduce the overhead of applying security patches.
  • Community and code review: A couple of sessions on optimising our working practices: most notably some clarifications to the maintainer role and how we can make code reviews more efficient.
  • Virtualization Modes: The next stage of PVH, which combines the best of HVM and PV. We also had discussions around some functionality that is currently developed in Linux on which PVH has dependencies.
  • Making Development more Scalable: A number of sessions to improve the toolstack and libxl. We covered topics such as making storage support pluggable via a plug-in architecture, making it easier to develop new PV drivers to support automotive and embedded vendors, and improvements to our build system, testing, stub domains and xenstored.
  • ARM support: There were a number of planning sessions for Xen ARM support. We covered the future roadmap, how to implement PCI passthrough, and how we can improve testing for the increasing range of ARM HW with support for virtualization, also applicable outside the server space.

There were many more sessions covering performance, scalability and other topics. The session’s host(s) post meeting notes on xen-devel@ (search for Hackathon in the subject line), if you want to explore any topic in more detail. To make it easier for people who do not follow our development lists, we also posted links to Hackathon related xen-devel@ discussions on our wiki.

Besides providing an opportunity to meet face-to-face, build bridges and solve problems, we always make sure that we have social events. After all Hackathons should be fun and bring people together. This year we had a dinner in Cambridge and of course the obligatory punting trip, which is part of every Cambridge trip.

Embarking on the punting journey

Embarking on the punting journey

Continued exploration of discovering the mysteries of the universe, while on a boat

Continued exploration of discovering the mysteries of the universe, while on a boat

Again, a big thanks to ARM for hosting the event! Also, a reminder that we’ll be hosting our Xen Project Developer Summit next August in Toronto, Canada. This event will happen directly after LinuxCon North America and is a great opportunity to learn more about Xen Project development and what’s happening within the Linux Foundation ecosystem at large. CFPs are still open until May 6th!

Stealthy monitoring with Xen altp2m

One of the core features that differentiates Xen from other open-source hypervisors is its native support for stealthy and secure monitoring of guest internals (aka. virtual machine introspection [1]). In Xen 4.6 which was was released last autumn several new features have been introduced that make this subsystem better; a cleaned-up, optimized API and ARM support being just some of the biggest items on this list. As part of this release of Xen, a new and unique feature was also successfully added by a team from Intel that make stealthy monitoring even better on Xen: altp2m. In this blog entry we will take a look at what it’s all about.

p2mIn Xen’s terminology, p2m stands for the memory management layer that handles the translation from guest physical memory to machine physical. This translation is critical for safely partitioning the real memory of the machine between Xen and the various VMs running as to ensure a VM can’t access the memory of another without permission. There are several implementations of this mechanism, including one with hardware support via Intel Extended Page Tables (EPT) available to HVM guests and PVH . In Xen’s terminology, this is called Hardware Assisted Paging (hap). In this implementation the hypervisor maintains a second pagetable, similar to the one in 64-bit operating systems use, dedicated to running the p2m translation. All open-source hypervisors that use this hardware assisted paging method use a single EPT per virtual machine to handle this translation, as most of the time the memory of the guest is assigned at VM creation and doesn’t change much afterwards.

altp2mXen altp2m is the first implementation which changes this setup by allowing Xen to create more then one EPT for each guest. Interestingly, the Intel hardware has been capable of maintaining up to 512 EPT pointers from the VMCS since the Haswell generation of CPUs. However, no hypervisor made use of this capability until now. This changed in Xen 4.6, where we can now create of up to 10 EPTs per guest. The primary reason for this feature is to use it with the #VE and VMFUNC extensions.

It can also be used by external monitoring applications via the Xen vm_event system.

Why alt2pm is a game-changer

Alt2pm is a game-changer for applications performing purely external monitoring is because it simplifies the monitoring process of multi-vCPU guests. The EPT layer has been successfully used in stealthy monitoring applications to track the memory accesses made by the VM from a safe vantage point by restricting the type of access the VM may perform on various memory pages. Since EPT permission violations trap into the hypervisor, the VM would receive no indication that anything out of the ordinary has happened. While the method allowed for stealthy tracing of R/W/X memory accesses of the guest, the memory permission needs to be relaxed in order to allow the guest to continue execution. When a single EPT is shared across multiple running vCPUs, relaxing the permissions to allow one vCPU to continue may inadvertently allow another one to perform the memory access we would otherwise want to track. While under normal circumstances such race-condition may rarely occur, malicious code could easily use this to hide some of its actions from a monitoring application.

Solutions to this problem exist already. For example we can pause all vCPUs while the one violating the access is single-stepped. This approach however introduces heavy overhead just to avoid a race-condition that may rarely occur in practice. Alternatively, one could emulate the instruction that was violating the EPT permission without relaxing the EPT access permissions, as Xen’s built-in emulator doesn’t use EPT to access the guest memory. This solution, while supported in Xen, is not particularly ideal either as Xen’s emulator is incomplete and is known to have issues that can lead to guest instability [2]. Furthermore, over the years emulation has been a hotbed of various security issues in many hypervisors (including Xen [3]), thus building security tools based on emulation is simply asking for trouble. It can be handy but should be used only when no other option is available.

Xen’s altp2m system changes this problem quite significantly. By having multiple EPTs we can have differing access permissions defined in each table, which can be easily swapped around by changing the active EPT index in the VMCS. When the guest makes a memory access that is monitored, instead of having to relax the access permission, Xen can simply switch to an EPT (called a view) that allows the operation to continue. Afterwards the permissive view can be switched back to the restricted view to continue monitoring. Since each vCPU has its own VMCS where this switching is performed, this monitoring can be performed specific to each vCPU, without having to pause any of the other ones, or having to emulate the access. All without the guest noticing any of this switching at all. A truly simple and elegant solution.

Other introspection methods for stealthy monitoring

EPT based monitoring is not the only introspecting technique used for stealthy monitoring. For example, the Xen based DRAKVUF Dynamic Malware Analysis [4] uses it in combination with an additional technique to maximum effect. The main motivation for that is because EPT based monitoring is known to introduce significant overhead, even with altp2m: the granularity of the monitoring is that of a memory page (4KB). For example, if the monitoring application is really just interested in when a function-entry point is called, EPT based monitoring creates a lot of “false” events when that page is accessed for the rest of the function’s code.

This can be avoided by enabling the trapping of debug instructions into the hypervisor, a built-in feature of Intel CPUs that Xen exposes to third-party applications. This method is used in DRAKVUF, which writes breakpoint instructions into the guests’ memory at code-locations of interest. Since we will only get an event for precisely the code-location we are interested in this method effectively reduces the overhead. However, the trade-off is that unlike EPT permissions the breakpoints are now visible to the guest. Thus, to hide the presence of the breakpoints from the guest, these pages need to get further protected by restricting the pages to be execute-only in the EPT. This allows DRAKVUF to remove the breakpoints before in-guest code-integrity checking mechanisms (like Windows Patchguard) can access the page. While with altp2m the EPT permissions can be safely used with multi-vCPU systems, using breakpoints similarly presents a race-condition: the breakpoint hit by one vCPU has to be removed to allow the guest to execute the instruction that was originally overwritten, potentially allowing another vCPU to do so as well without notice.

altp2m-shadowFortunately, altp2m has another neat feature that can be used to solve this problem. Beside allowing for changing the memory permissions in the different altp2m views, it also allows to change the mapping itself! The same guest physical memory can be setup to be backed by different pages in the different views. With this feature we can really think of guest physical memory as “virtual”: where it is mapped really depends on which view the vCPU is running on. Using this feature allows us to hide the presence of the breakpoints in a brand new way. To do this, first we create a complete shadow copy of the memory page where a breakpoint is going to be written and only write the breakpoint into this shadow copy. Now, using altp2m, we setup a view where the guest physical memory of the page gets mapped to our shadow copy. The guest continues to access its physical memory as before, but underneath it is now using the trapped shadow copy. When the breakpoint is hit, or if something is trying to scan the code, we simply switch the view to the unaltered view for the duration of a single-step, then switch back to the trapped view. This allows us to hide the presence of the breakpoints specific to each vCPU! All without having pause any of the other vCPUs or having to emulate. The first open-source implementation of this tracing has been already merged into the DRAKVUF Malware Analysis System and is available as a reference implementation for those interested in more details.

Conclusion

As we can see, Xen continues to be on the forefront of advancing the development of virtualization based security application and allowing third-party tools to create some very exotic setups. This flexibility is what’s so great about Xen and why it will continue to be a trend-setter for the foreseeable future

References

[1] Virtual Machine Introspection
[2] xen-devel@: Failed vm entry with heavy use of emulator
[3] Hardening Hypervisors Against VENOM-Style Attacks
[4] DRAKVUF Malware Analysis System (drakvuf.com)
[5] Stealthy, Hypervisor-based Malware Analysis (Presentation)

Xen Project 4.5.3 Maintenance Release is Available

I am pleased to announce the release of Xen 4.5.3. Xen Project Maintenance releases are released in line with our Maintenance Release Policy. We recommend that all users of the 4.5 stable series update to this point release.

Xen 4.5.3 is available immediately from its git repository:

    xenbits.xenproject.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.5
    (tag RELEASE-4.5.3)

or from the Xen Project download page at www.xenproject.org/downloads/xen-archives/xen-45-series/xen-453.html.

This release contains many bug fixes and improvements. For a complete list of changes in this release, please check the lists of changes on the download page.

We recommend all users of the 4.5 stable series to update to this latest point release.