Q&A: Xen Project Release Strengthens Security and Pushes New Use Cases

The following Q&A with Lars Kurth, the Xen Project chairperson, was first published on Linux.com.

Xen Project technology supports more than 10 million users and is a staple in some of the largest clouds in production today, including Amazon Web Service, Tencent, and Alibaba’s Aliyun. Recently, the project announced the arrival of Xen Project Hypervisor 4.7. This new release focuses on improving code quality, security hardening and features, and support for the latest hardware. It is also the first release of the project’s fixed-term June – December release cycles. The fixed-term release cycles provide more predictability making it easier for consumers of Xen to plan ahead.

We recently sat down with the Xen Project chairperson, Lars Kurth, to talk about some of the key features of the release and the future of Xen Project technology. Lars will be discussing this topic and more during Xen Project’s Developer Summit in Toronto, CA from August 25-26 — the conference is directly after LinuxCon North America.

Q: What was the focus on this release?

Lars Kurth: There were five areas that we focused on for this release (full details are in our blog). In summary, we focused on security features, migration support, performance and workloads, support for new hardware features, and drivers and devices (Linux, FreeBSD and other).

Security is consistently something that we focus on in all of our releases. There are a lot of people that rely on Xen Project technology and security is our top concern in any release as well as how we organize our process around security disclosures.

Q: What was the biggest feature coming out of this release?

Lars: The biggest feature for us is live patching, which is a technology that enables re-boot free deployment for security patches to minimize disruption and downtime during security upgrades for cloud admins. It essentially eliminates all cloud reboots, making cloud providers and their users much more safe. It also eliminates a lot of headaches for system and DevOps admins of the world.

Q: Xen is often associated with the cloud, but are there additional use cases that you see growing around this technology, if so why?

Lars: We are seeing a lot of growth in terms of contributions, as well as many different use cases emerging, including automotive, aviation, embedded scenarios, security, and also IoT. In addition, we continue to grow within the public cloud sector and traditional server virtualization.

On the security front, for example, a number of vendors such as A1Logic, Bitdefender, Star Lab and Zentific have released or are working on new Xen Project-based security solutions. In addition, the security focused and Xen-based OpenXT project has started to work more closely with the Xen Project community.

Long-time contributors to the Xen Project, such as DornerWorks – a premier provider of electronic engineering services for the aerospace, medical, automotive, and industrial markets – have expanded their scope and are now providing support for the Xen Xilinx Zynq Distribution targeting embedded use-cases. We have also seen an increasing number of POCs and demos of automotive solutions, which include Xen as a virtualization solution.

Growth in these sectors is largely due to the Xen Project’s flexibility, extensibility, customisability and a clear lead when it comes to security-related technologies. Over the last year, we have also seen contributions increase from developers with strong security and embedded backgrounds. In fact, this totaled nearly 17 percent of the overall contributions in this release cycle, up from 9 percent in the previous release.

Q: How did you address these uses cases in this latest release?

Lars: We introduced the ability to remove core Xen Project Hypervisor features at compile via KCONFIG. This creates a more lightweight hypervisor and eliminates extra attack surfaces that are beneficial in security-first environments and microservice architectures. Users will still be able to get the core hypervisor functions, but they won’t receive all the drivers, schedulers, components or features that might not fit their use case.

Essentially it gives people an “a la carte” feature set. They can decide what they need for compliance, safety or performance reasons.

Q: Were there any new contributors for this release that surprised you?

Lars: We had three new companies contributing to the project: Star Lab, Bosch and Netflix. I met engineers from Star Lab for the first time at the 2015 Developer Summit less than a year ago, and helped introduce them to the Project’s culture. In that short period of time, Doug Goldstein from Star Lab has moved into the top five contributors and top 10 code reviewers for the Project.

I was surprised about Netflix’s contributions; I didn’t even know the company used Xen. Netflix improved and secured the VPMU feature, which is incredibly useful for system tuning and performance monitoring. Bosch Car Multimedia GmbH added some new ARM functionality. In addition, we have seen quite a bit of Xen related development in upstream and downstream projects such as Linux, FreeBSD, NetBSD, OpenBSD, QEMU and Libvirt.

Q: What’s next for Xen Project? Where do you think the technology is heading in the future and why?

Lars: In the last three releases, we introduced several major new features such as PVH, COLO, new schedulers, VMI, Live Patching, Graphics Virtualization, etc. and significant re-work of existing features such as Migration and the Xen Security Modules (XSM). Looking at trends within the community, I expect that stepwise evolution of large new features to continue.

Some new capabilities, such as restartable Dom0’s, and additional techniques to provide more isolation and security, are also likely to appear. In addition, it looks likely that we will see some GPU virtualization capabilities for GPUs that target the ARM ecosystem, although it is not yet clear whether these will be available as open source. I also expect that both Intel and ARM hardware features will be closely tracked.

Some areas, such as new schedulers, XSM, PVH and Live Patching, will see significant efforts to harden and improve existing functionality. The goal is to ensure their swift adoption in commercial products and Linux and BSD distributions. Some features, which are not enabled by default are likely to become part of the Xen Project Hypervisor’s default configuration.

Xen Project 4.7 and 4.6.3 Release

I’m pleased to announce the release of Xen Project Hypervisor 4.7 and Xen Project Hypervisor 4.6.3.

Xen Project Hypervisor 4.7

This new release focuses on improving code quality, security hardening, security features, live migration support, usability improvements and support for new hardware features — this is also the first release of our fixed term June – December release cycle.

We continue to strive to make Xen Project Hypervisor the most secure open source hypervisor to match the security challenges in cloud computing, and for embedded and IoT use-cases. We are continuing to improve upon the performance and scalability for our users, and aim to continuously bring many new features to our users in a timely manner.

To make it easier to understand the major changes during this release cycle, I’ve grouped them below into several categories:

  • Security Features
  • Migration Support
  • Performance and Workloads
  • Support for new Hardware Features
  • Drivers and Devices (Linux, FreeBSD and other)

Security Features

Reboot-free Live Patching: Xen Project Hypervisor 4.7 comes equipped with Live Patching, a technology that enables re-boot free deployment of security patches to minimize disruption and downtime during security upgrades for system administrators and DevOps practitioners. Xen Project 4.7 implements version 1 of the Xen Project’s Live Patching specification, which is designed to encode the vast majority of security patches (approximately 90%) as Live Patching payloads. This version ships with a Live Patching enabled hypervisor and payload deployment tools and is available as a technology preview.

KCONFIG support: For security, embedded automotive and IoT use cases, Xen Project introduced the ability to remove core Xen Hypervisor features at compile time via KCONFIG. This ability creates a more lightweight hypervisor and eliminates extra attack surfaces that are beneficial in security-first environments, microservice architectures and environments that have heavy compliance and certification needs, like automotive.

Improvements to the Virtual Machine Introspection (VMI) subsystem: A number of performance, scalability, robustness and interface improvements have been added to the Virtual Machine Introspection subsystem, that was introduced in Xen 4.5. In addition, Bitdefender Hypervisor Introspection leveraging Xen Project Virtual Machine Introspection, has recently been released as a new enterprise security solution to discover and remedy deep threats that remain hidden via traditional endpoint security tools.

Foundation work to tolerate a restartable Dom0: Several key components in a Xen Project system run in Dom0, which make Dom0 the single point of failure. Xen Project has been able to run xenstored, the daemon for managing the hypervisor’s central settings repository on a Xen Project host, in a sandboxed Virtual Machine called xenstored stub domain since Xen Project version 4.2. In Xen 4.7, we have made it easier to build xenstored stub domains and for them to tolerate a Dom0 restart. This will make Dom0 less critical to a Xen Project system and help us move towards a more robust and secure architecture in the future. More work in this area is expected in subsequent releases.

Migration Support

Improved Migration support: CPU ID Levelling enables migration of VM’s between a larger range of non-identical hosts than previously supported.

Fault Tolerance / Coarse-grained Lock-stepping (COLO): Xen 4.5 laid the foundation for COLO while improving the Xen Project’s Hypervisors Live Migration and Remus High Availability support. The COLO Manager, which introduces a relaxed approach to checkpointing that avoids unnecessary checkpoints enabling near native performance for many workloads, has been fully integrated as an experimental feature into Xen 4.7. Note that the COLO Block Replication and COLO Proxy components, both of which are QEMU components, are currently still reviewed by the QEMU community. Both components are available as out-of-tree add-ons to the Xen Project Hypervisor, until fully integrated into QEMU.

Performance and Workloads

Support for a wider range of workloads and applications: The PV guest limit restriction of 512GB has been removed to allow the creation of huge PV domains in the TB range. TB sized VMs, coupled with Xen Project’s existing support for 512 vCPUs per VM, enable execution of memory and compute intensive workloads such as big data analytics workloads and in-memory databases.

Improved Credit 2 scheduler: The Credit2 scheduler is one (big) step closer to being ready for production use. It is now possible to instruct the scheduler to organize its runqueues and perform load balancing at core, socket or NUMA node granularity. More fine grained (core) configurations, deliver more aggressive load balancing, and are best suited for medium size systems. This feature has been proven to enable very good performance, especially if Hyper Threading is present.

Less fine grained configurations entail less overhead, and is suitable for larger servers or when no Hyper Threading is available. In addition, Credit2 has been extended to allow pinning of vCPUs to pCPUs (also known as “hard affinity”), allowing system administrators to configure the system in the exact way they want, and achieve the best setup for a given workload (for instance, a guarantee that a certain subset of vCPUs are always able to run when they need to run).

Improved RTDS scheduler: The RTDS scheduler is a real-time CPU scheduler built to provide guaranteed CPU capacity to guest VMs on SMP hosts, which primarily targets embedded, real-time and low-latency workloads. In Xen Project 4.7, the scheduling model has been changed from a quantum-driven to an event-driven model, which reduces scheduling overhead and thus scalability and performance for embedded and realtime workloads. In addition, per-VCPU parameter configuration has been added to allow better scheduler control for specialised workloads.

Per-cpu reader-writer lock: This new infrastructure allows for the fast path read case to have low overhead by only setting/clearing a per-cpu variable for using the read lock. After transforming various hypervisor locks to this infrastructure, VM-VM network transfer with 16 queues jumped from 15 gbit/s to 48 gbit/s on a 2 socket Haswell-EP host.

Usability Improvements

PVUSB Support: In Xen Project 4.7, a new XL command line interface to manage PVUSB devices has been introduced to manage PVUSB devices for PV guests. Both in kernel PVUSB backend and QEMU backend are supported.

Hot plugging of QEMU disk backends: Xen Project now enables hot-plugging of USB devices as well as QEMU disk backends, such as drbd, iscsi, and more in HVM guests. This new feature allows users to add and remove disk backends to virtual machines without the need to reboot the guest.

Soft-reset: The soft reset feature for HVM guests allows for a more graceful shutdown and restart of the HVM guest.

New Hardware Support

Features specific to the ARM Architecture

SBBR Compliance: Xen Project now supports booting on hosts that expose ACPI 6.0 (and later) information. The ARM Server Base Boot Requirements (SBBR) stipulate that compliant systems need to express hardware resources with ACPI; thus this support will come in useful for ARM Servers. This effort was carried out by Shannon Zhao of Linaro with minor patches from Julien Grall of ARM.

PCSI 1.0 Compatibility: PSCI 1.0 compatibility allows Xen Project software to operate on systems that expose PSCI 1.0 methods. Now, all 1.x versions of PSCI will be compatible with Xen Project software. More information on Power State Coordination Interface can be found here. This effort was also carried out by Julien Grall with a patch from Dirk Behme of Bosch.

vGIC-v3: Virtual Generic Interrupt Controller version 3. Reworked to be spec-compliant and optimised in some code paths.

Wallclock support: ARM guest can now get wallclock time directly from Xen Project via shared info page.

Features specific to Intel® Xeon® processor product family

Improved Interrupt Efficiency: Xen Project 4.7 supports VT-d Posted Interrupts, which provides hardware-level acceleration to increase interrupt virtualization efficiency. It reduces latency and improves user experience through performance improvements, especially for interrupt-intensive front-end workloads such as web servers. Note that Posted Interrupts in Xen Project 4.7 are still experimental and disabled by default.

Code and Data Prioritization: Xen Project 4.7 is the first to include Code and Data Prioritization (CDP), part of the Intel® Resource Director Technology (RDT) Framework and an extension of Cache Allocation Technology (CAT), first introduced in Xen Project 4.6. The introduction of CDP allows isolation of code/data within the shared L3 cache of multi-tenant environments, reducing contention and improving performance.

Other Intel Features: Additional features specific to the Intel Xeon processor family in Xen Project 4.7 include: VMX TSC Scaling, which allows for easier migration between machines with different CPU frequencies and support for Memory Protection Keys, a new security feature for hardening the software stack.

Drivers and Devices (Linux, FreeBSD and other)

During the Xen Project 4.7 release cycle, we made significant improvements to major operating systems and components we rely on to improve interoperability. During this development cycle 1494 Xen Project only related changesets – mostly bug fixes and small improvements – were applied to Linux, FreeBSD, NetBSD, QEMU and the Windows PV drivers: more than twice as many as in the 4.6 release cycle.


With dozens of major improvements, many more bug fixes and small improvements, and significant improvements to Drivers and Devices, Xen Project 4.7 reflects a thriving community around the Xen Project Hypervisor.

We are extremely proud of achieving the highest quality of the release while increasing development velocity across the hypervisor and its upstream dependencies by about 16%. In particular, our latest security related features enable Xen Project software to compete in the security appliance market and help answer some of the difficult questions regarding security in the cloud era.

We set out at the beginning of this release cycle to foster greater collaboration among vendors, individual developers, upstream maintainers, other projects and distributions. During this release cycle we continued to see an increasing influx of patches and newcomers such as Star Lab, Bosch and Netflix. We had a significant amount of contributions from cloud providers, software vendors, hardware vendors, academic researchers and individuals to help with this release. Major contributors for this particular release come from Citrix, SUSE, Intel, Star Lab, Oracle, Linaro, Fujitsu, Bitdefender, Red Hat, Huawei, ARM, Novetta, Broadcom, Xilinx, Bosch, AMD, GlobalLogic, NSA, Netflix and a number of universities and individuals. Thank you to all who participated.

As the release manager, I would like to thank everyone for their contributions (either in the form of patches, bug reports or packaging efforts) to the Xen Project. This release wouldn’t have happened without contributions from so many people around the world. Please check out our 4.7 contributor acknowledgement page.

The source can be located in the http://xenbits.xen.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.7 tree (tag RELEASE-4.7.0) or can be downloaded as tarball from our website. More information can be found at

Xen Project Hypervisor 4.6.3

The Xen Project 4.6.3 release is a maintenance release which comprises bug fixes and security updates. This is release is available immediately from its git repository
(tag RELEASE-4.6.3) or from the Xen Project download page
(where a list of changes can also be found).

We recommend all users of the 4.6 stable series which do not wish to upgrade to Xen 4.7, to update to this latest point release.

Note regarding version numbering: an issue was found late in the release process,
after one of the affected qemu trees was already tagged with a signed release git tag. Signed git tags provide a secure way of accounting for the source code, but once created they cannot be removed. Thus, the project could have released this maintenance release with a known issue, or fix the issue and skip a version number. We opted for the latter and decided to skip version 4.6.2.

Xen Project Community Hosts Annual Developers Summit in August

We recently announced the program and speakers for our Xen Project Developer Summit happening in Toronto, Canada from August 25-26, 2016. The event will be co-located with LinuxCon North America.

The Xen Project hypervisor powers the new needs of computing and virtualization through a rich ecosystem of community members that focus on everything from security, embedded, and web-scale environments. The Summit is an opportunity for developers and software engineers to collaborate and discuss the latest advancements of Xen Project software, and better understand what’s next for Xen Project technology, virtualization and cloud computing.

In addition to presentations, we will be running a half-day hackathon alongside the Summit on the last day. Xen Project hackathons have evolved in format into a series of structured problem-solving sessions that scale up to 50 people.

This flagship event features presentations on the latest developments, best practices, collaboration, product roadmap updates and future planning from developers and users who are leading the way in server density, hardware, automotive, cloud and enterprise security. To view the full schedule and register, please head here: http://events.linuxfoundation.org/events/xen-project-developer-summit/program/schedule

This event is being sponsored by Citrix (Diamond sponsor), Huawei (Platinum sponsor) and Intel (Platinum sponsor). Please be sure to follow updates on the event via Xen Project’s Twitter, Google+ or Facebook page. Hashtag for the event is #xendevsummit.

We look forward to seeing you there!

Introducing the Xen Project Code Review Dashboard

The Xen Project’s code contributions have grown more than 10% each year. Although growth is extremely healthy to the project as a whole, it has its growing pains. For the Xen Project, it led to issues with its code review process: maintainers believed that their review workload increased and a number of vendors claimed that it took significantly longer for contributions to be upstreamed, compared to the past.

The project developed some basic scripts that correlated development list traffic with git commits, which showed indeed that it took longer for patches to be committed. In order to identify possible root causes, the project initially ran a number of surveys to identify possible causes for the slow down. Unfortunately, many of the observations made by community members contradicted each other, and were thus not actionable. To solve this problem, the Xen Project worked with Bitergia, a company that focuses on analyzing community software development processes, to better understand and address the issues at hand. We worked with Bitergia on an initial statistical analysis of the code review process and later on a Code Review Dashboard for use by the community. The following OSCON presentation lays out the journey, the project went through:

Findings of the Initial Code Review Study

There were three key areas that we found that were causing the slow down:

  • Huge growth in comment activity from 2013 to 2015
  • We also saw that the time it took to merge patches (=time to merge) increased significantly from 2012 to the first half of 2014. However, from the second half of 2014 time to merge moved back to its long term average. This was a strong indicator that the pre-emptive measures we took, such as a focus on design and architecture reviews and contributor training, actually had an effect.
  • Looking at this in more detail, it turned out that complex patches were taking significantly longer to merge than small patches. As it turns out, a significant number of new features were actually rather complex. At the same time, the demands on the project to deliver better quality and security had also raised the bar for what could be accepted, which impacted new contributors more than established ones.

Introducing the Xen Project Code Review Dashboard

To make the tooling that we developed more accessible to the entire community, the Xen Project Advisory Board funded the development of a Code Review Dashboard. We defined a set of use-cases and supporting data that broadly covered three areas:

  • Community use cases to encourage desired behaviour: this would be metrics such as real review contributions (not justed ACKed-by and Reviewed-by flags), comparing review activity against contributions.
  • Performance use cases that would allow us to spot issues early: these would allow us to filter time related metrics by a number of different criteria such as complexity of a patch series.
  • Backlog use cases to optimize process and focus. The intention here was to give contributors and maintainers tools to see what reviews are active, nearly complete, complete or stale.

To find out more check out

Value for other projects

Like many FOSS projects, the Xen Project code review process uses a mailing list-based review process, and this could be a good blueprint for projects that are finding themselves in the same predicament. It is already clear, that there are many useful additions that can in future be added to the technology we have developed. In addition, we are currently working on improvements of the Code Review Dashboard as part of an Outreachy Project by Priya V (check out her blog), which is jointly mentored by Lars Kurth (Xen Project) and Jesus M. Gonzalez-Barahona (Bitergia). All the code is open source: if you want to have a look, check out the code and the contribution guide.

Intel hosts OpenXT Summit on Xen Project based Client Virtualization, June 7-8 in Fairfax, VA, USA

This is a guest blog post by Rich Persaud, former member of the Citrix XenServer and XenClient engineering and business teams. He is currently a consultant to BAE Systems, working on the OpenXT project, which stands on the shoulders of the Xen Project, OpenEmbedded Linux and XenClient XT.

While the Xen Project is well known for servers and hosted infrastructure, Type-1 hypervisors have been used in client endpoints and network appliances, improving security and remote manageability. Virtualization-based security in Qubes and Windows 10 is also educating system administrators about hardware security (IOMMU and TPM) and application trust models.

Released as open-source software in 2014, OpenXT is a development toolkit for hardware-assisted security research and appliance integration. It includes hardened Linux VMs that can be configured as a user-facing software appliance for client devices, with virtualization of storage, network, input, sound, display and USB devices. Hardware targets include laptops, desktops and workstations.

OpenXT stands on the shoulders of the Xen Project, OpenEmbedded Linux and XenClient XT. It is optimized for hardware-assisted virtualization with an IOMMU and a TPM. It configures Xen network driver domains, Linux stub domains, Xen Security Modules, Intel TXT, SE Linux, GPU passthrough and VPNs. Guest operating systems include Windows, Linux and FreeBSD. VM storage options include encrypted VHD files with boot-time measurement and non-persistence.

The picture below shows a typical OpenXT software stack, including Xen, Linux and other components.

The picture above shows one of many configurations of the OpenXT software stack, including Xen, Linux and other components.

OpenXT enables loose coupling of open-source and proprietary software components, verifiable measurements of hardware and software, and verified launch of derivative products. It has been used to develop locally/centrally managed software appliances that isolate high-risk workloads, networks and devices.

The inaugural OpenXT Summit brings together developers and ecosystem participants for a 2-day conference in Fairfax, VA, USA on June 7-8, 2016. The event is hosted by Intel Corporation. The audience for this event includes kernel and application developers, hardware designers, system integrators and security architects.

The 2016 OpenXT Summit will chart the evolution of OpenXT from cross-domain endpoint virtualization to an extensible systems innovation platform, enabling derivative products to make security assurances for diverse hardware, markets and use cases.

The Summit includes one day of presentations, a networking reception and one day of moderated technical discussions. Presentation topics will include OpenXT architecture, TPM 2.0, Intel SGX, Xen security, measured launch, graphics virtualization and NSA research on virtualization and trusted computing.

For more information, please see the event website at http://openxt.org/summit.

For presentations and papers related to OpenXT, please see http://openxt.org/history.

Announcing Xen Project 4.7 RC and Test Day Schedule

Yesterday we created Xen 4.7 RC2 and will release a new release candidate every Wednesday, until we declare a release candidate as the final candidate and cut the Xen 4.7 release. We will also hold a Test Day every Friday for the release candidate that was released the Wednesday prior to the Test Day. This means we will have Test Days on May 13th, 20th, 27th and June 3rd. Your testing is still valuable on other days, so please feel free to send Test Reports as outlined below at any time.

Getting, Building and Installing a Release Candidate

Release candidates are available from our git repository at

git://xenbits.xen.org/xen.git (tag 4.7.0-<rc>)

where <rc> is rc1, rc2, rc3, etc. and as tarball from


Detailed build and Install instructions can be found on the Test Day Wiki.

Testing new Features, Test and Bug Reports

You can find Test Instructions for new features on our Test Day Wiki and instructions for general tests on Testing Xen. The following pages provide information on how to report successful tests and how to report bugs and issues.

Happy Testing!