Tag Archives: 4.7.0

Xen Project 4.7 and 4.6.3 Release

I’m pleased to announce the release of Xen Project Hypervisor 4.7 and Xen Project Hypervisor 4.6.3.

Xen Project Hypervisor 4.7

This new release focuses on improving code quality, security hardening, security features, live migration support, usability improvements and support for new hardware features — this is also the first release of our fixed term June – December release cycle.

We continue to strive to make Xen Project Hypervisor the most secure open source hypervisor to match the security challenges in cloud computing, and for embedded and IoT use-cases. We are continuing to improve upon the performance and scalability for our users, and aim to continuously bring many new features to our users in a timely manner.

To make it easier to understand the major changes during this release cycle, I’ve grouped them below into several categories:

  • Security Features
  • Migration Support
  • Performance and Workloads
  • Support for new Hardware Features
  • Drivers and Devices (Linux, FreeBSD and other)

Security Features

Reboot-free Live Patching: Xen Project Hypervisor 4.7 comes equipped with Live Patching, a technology that enables re-boot free deployment of security patches to minimize disruption and downtime during security upgrades for system administrators and DevOps practitioners. Xen Project 4.7 implements version 1 of the Xen Project’s Live Patching specification, which is designed to encode the vast majority of security patches (approximately 90%) as Live Patching payloads. This version ships with a Live Patching enabled hypervisor and payload deployment tools and is available as a technology preview.

KCONFIG support: For security, embedded automotive and IoT use cases, Xen Project introduced the ability to remove core Xen Hypervisor features at compile time via KCONFIG. This ability creates a more lightweight hypervisor and eliminates extra attack surfaces that are beneficial in security-first environments, microservice architectures and environments that have heavy compliance and certification needs, like automotive.

Improvements to the Virtual Machine Introspection (VMI) subsystem: A number of performance, scalability, robustness and interface improvements have been added to the Virtual Machine Introspection subsystem, that was introduced in Xen 4.5. In addition, Bitdefender Hypervisor Introspection leveraging Xen Project Virtual Machine Introspection, has recently been released as a new enterprise security solution to discover and remedy deep threats that remain hidden via traditional endpoint security tools.

Foundation work to tolerate a restartable Dom0: Several key components in a Xen Project system run in Dom0, which make Dom0 the single point of failure. Xen Project has been able to run xenstored, the daemon for managing the hypervisor’s central settings repository on a Xen Project host, in a sandboxed Virtual Machine called xenstored stub domain since Xen Project version 4.2. In Xen 4.7, we have made it easier to build xenstored stub domains and for them to tolerate a Dom0 restart. This will make Dom0 less critical to a Xen Project system and help us move towards a more robust and secure architecture in the future. More work in this area is expected in subsequent releases.

Migration Support

Improved Migration support: CPU ID Levelling enables migration of VM’s between a larger range of non-identical hosts than previously supported.

Fault Tolerance / Coarse-grained Lock-stepping (COLO): Xen 4.5 laid the foundation for COLO while improving the Xen Project’s Hypervisors Live Migration and Remus High Availability support. The COLO Manager, which introduces a relaxed approach to checkpointing that avoids unnecessary checkpoints enabling near native performance for many workloads, has been fully integrated as an experimental feature into Xen 4.7. Note that the COLO Block Replication and COLO Proxy components, both of which are QEMU components, are currently still reviewed by the QEMU community. Both components are available as out-of-tree add-ons to the Xen Project Hypervisor, until fully integrated into QEMU.

Performance and Workloads

Support for a wider range of workloads and applications: The PV guest limit restriction of 512GB has been removed to allow the creation of huge PV domains in the TB range. TB sized VMs, coupled with Xen Project’s existing support for 512 vCPUs per VM, enable execution of memory and compute intensive workloads such as big data analytics workloads and in-memory databases.

Improved Credit 2 scheduler: The Credit2 scheduler is one (big) step closer to being ready for production use. It is now possible to instruct the scheduler to organize its runqueues and perform load balancing at core, socket or NUMA node granularity. More fine grained (core) configurations, deliver more aggressive load balancing, and are best suited for medium size systems. This feature has been proven to enable very good performance, especially if Hyper Threading is present.

Less fine grained configurations entail less overhead, and is suitable for larger servers or when no Hyper Threading is available. In addition, Credit2 has been extended to allow pinning of vCPUs to pCPUs (also known as “hard affinity”), allowing system administrators to configure the system in the exact way they want, and achieve the best setup for a given workload (for instance, a guarantee that a certain subset of vCPUs are always able to run when they need to run).

Improved RTDS scheduler: The RTDS scheduler is a real-time CPU scheduler built to provide guaranteed CPU capacity to guest VMs on SMP hosts, which primarily targets embedded, real-time and low-latency workloads. In Xen Project 4.7, the scheduling model has been changed from a quantum-driven to an event-driven model, which reduces scheduling overhead and thus scalability and performance for embedded and realtime workloads. In addition, per-VCPU parameter configuration has been added to allow better scheduler control for specialised workloads.

Per-cpu reader-writer lock: This new infrastructure allows for the fast path read case to have low overhead by only setting/clearing a per-cpu variable for using the read lock. After transforming various hypervisor locks to this infrastructure, VM-VM network transfer with 16 queues jumped from 15 gbit/s to 48 gbit/s on a 2 socket Haswell-EP host.

Usability Improvements

PVUSB Support: In Xen Project 4.7, a new XL command line interface to manage PVUSB devices has been introduced to manage PVUSB devices for PV guests. Both in kernel PVUSB backend and QEMU backend are supported.

Hot plugging of QEMU disk backends: Xen Project now enables hot-plugging of USB devices as well as QEMU disk backends, such as drbd, iscsi, and more in HVM guests. This new feature allows users to add and remove disk backends to virtual machines without the need to reboot the guest.

Soft-reset: The soft reset feature for HVM guests allows for a more graceful shutdown and restart of the HVM guest.

New Hardware Support

Features specific to the ARM Architecture

SBBR Compliance: Xen Project now supports booting on hosts that expose ACPI 6.0 (and later) information. The ARM Server Base Boot Requirements (SBBR) stipulate that compliant systems need to express hardware resources with ACPI; thus this support will come in useful for ARM Servers. This effort was carried out by Shannon Zhao of Linaro with minor patches from Julien Grall of ARM.

PCSI 1.0 Compatibility: PSCI 1.0 compatibility allows Xen Project software to operate on systems that expose PSCI 1.0 methods. Now, all 1.x versions of PSCI will be compatible with Xen Project software. More information on Power State Coordination Interface can be found here. This effort was also carried out by Julien Grall with a patch from Dirk Behme of Bosch.

vGIC-v3: Virtual Generic Interrupt Controller version 3. Reworked to be spec-compliant and optimised in some code paths.

Wallclock support: ARM guest can now get wallclock time directly from Xen Project via shared info page.

Features specific to Intel® Xeon® processor product family

Improved Interrupt Efficiency: Xen Project 4.7 supports VT-d Posted Interrupts, which provides hardware-level acceleration to increase interrupt virtualization efficiency. It reduces latency and improves user experience through performance improvements, especially for interrupt-intensive front-end workloads such as web servers. Note that Posted Interrupts in Xen Project 4.7 are still experimental and disabled by default.

Code and Data Prioritization: Xen Project 4.7 is the first to include Code and Data Prioritization (CDP), part of the Intel® Resource Director Technology (RDT) Framework and an extension of Cache Allocation Technology (CAT), first introduced in Xen Project 4.6. The introduction of CDP allows isolation of code/data within the shared L3 cache of multi-tenant environments, reducing contention and improving performance.

Other Intel Features: Additional features specific to the Intel Xeon processor family in Xen Project 4.7 include: VMX TSC Scaling, which allows for easier migration between machines with different CPU frequencies and support for Memory Protection Keys, a new security feature for hardening the software stack.

Drivers and Devices (Linux, FreeBSD and other)

During the Xen Project 4.7 release cycle, we made significant improvements to major operating systems and components we rely on to improve interoperability. During this development cycle 1494 Xen Project only related changesets – mostly bug fixes and small improvements – were applied to Linux, FreeBSD, NetBSD, QEMU and the Windows PV drivers: more than twice as many as in the 4.6 release cycle.


With dozens of major improvements, many more bug fixes and small improvements, and significant improvements to Drivers and Devices, Xen Project 4.7 reflects a thriving community around the Xen Project Hypervisor.

We are extremely proud of achieving the highest quality of the release while increasing development velocity across the hypervisor and its upstream dependencies by about 16%. In particular, our latest security related features enable Xen Project software to compete in the security appliance market and help answer some of the difficult questions regarding security in the cloud era.

We set out at the beginning of this release cycle to foster greater collaboration among vendors, individual developers, upstream maintainers, other projects and distributions. During this release cycle we continued to see an increasing influx of patches and newcomers such as Star Lab, Bosch and Netflix. We had a significant amount of contributions from cloud providers, software vendors, hardware vendors, academic researchers and individuals to help with this release. Major contributors for this particular release come from Citrix, SUSE, Intel, Star Lab, Oracle, Linaro, Fujitsu, Bitdefender, Red Hat, Huawei, ARM, Novetta, Broadcom, Xilinx, Bosch, AMD, GlobalLogic, NSA, Netflix and a number of universities and individuals. Thank you to all who participated.

As the release manager, I would like to thank everyone for their contributions (either in the form of patches, bug reports or packaging efforts) to the Xen Project. This release wouldn’t have happened without contributions from so many people around the world. Please check out our 4.7 contributor acknowledgement page.

The source can be located in the http://xenbits.xen.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.7 tree (tag RELEASE-4.7.0) or can be downloaded as tarball from our website. More information can be found at

Xen Project Hypervisor 4.6.3

The Xen Project 4.6.3 release is a maintenance release which comprises bug fixes and security updates. This is release is available immediately from its git repository
(tag RELEASE-4.6.3) or from the Xen Project download page
(where a list of changes can also be found).

We recommend all users of the 4.6 stable series which do not wish to upgrade to Xen 4.7, to update to this latest point release.

Note regarding version numbering: an issue was found late in the release process,
after one of the affected qemu trees was already tagged with a signed release git tag. Signed git tags provide a secure way of accounting for the source code, but once created they cannot be removed. Thus, the project could have released this maintenance release with a known issue, or fix the issue and skip a version number. We opted for the latter and decided to skip version 4.6.2.

Mirage OS v2.0: The new features

The first release of Mirage OS back in December 2013 introduced the prototype of the unikernel concept, which realised the promise of a safe, flexible mechanism to build highly optimized software stacks purpose-built for deployment in the public cloud (see the overview of Mirage OS for some background). Since then, we’ve been hard at work using and extending Mirage for real projects and the community has been steadily growing.

Today, we’re thrilled to announce the release of Mirage OS v2.0! Over the past few weeks the team has been hard at work writing about all the new features in this latest release, which I’ve been busy co-ordinating. Below are summaries of those features and links to in-depth blog posts where you can learn more:

Thomas Leonard's Cubieboard2

Thomas Leonard’s Cubieboard2

ARM device support: While the first version of Mirage was specialised towards conventional x86 clouds, the code generation and boot libraries have now been made portable enough to operate on low-power embedded ARM devices such as the Cubieboard 2. This is a key part of our efforts to build a safe, unified multiscale programming model for both cloud and mobile workloads as part of the Nymote project. We also upstreamed the changes required to the Xen Project so that other unikernel efforts like HalVM or ClickOS can benefit.

Irmin – distributed, branchable storage: Unikernels usually execute in a distributed, disconnection-prone environment (particularly with the new mobile ARM support). We therefore built the Irmin library to explicitly make synchronization easier via a Git-like persistence model that can be used to build and easily trace the operation of distributed applications across all of these diverse environments.

OCaml TLS: The philosophy of Mirage is to construct the entire operating system in a safe programming style, from the device drivers up. This continues in this release with a comprehensive OCaml implementation of Transport Layer Security, the most widely deployed end-to-end encryption protocol on the Internet (and one that is very prone to bad security holes). The series of posts is written by Hannes Mehnert and David Kaloper.

Modularity and communication: Mirage is built on the concept of a library operating system, and this release provides many new libraries to flexibly extend applications with new functionality.

  • Fitting the modular Mirage TCP/IP stack together” by Mindy Preston explains the rather unique modular architecture of our TCP/IP stack that lets you swap between the conventional Unix sockets API, or a complete implementation of TCP/IP in pure OCaml.
  • Vchan: low-latency inter-VM communication channels” by Jon Ludlam shows how unikernels can communicate efficiently with each other to form distributed clusters on a multicore Xen host, by establishing shared memory rings with each other.
  • Modular foreign function bindings” by Jeremy Yallop continues the march towards abstraction by expaining how to interface safely with code written in C, without having to write any unsafe C bindings! This forms the basis for allowing Xen unikernels to communicate with existing libraries that they may want to keep at arm’s length for security reasons.

All the libraries required for these new features are regularly released into the OPAM package manager, so just follow the installation instructions to give them a spin. A release this size probably introduces minor hiccups that may cause build failures, so we very much encourage bug reports on our issue tracker or questions to our mailing lists. Don’t be shy: no question is too basic, and we’d love to hear of any weird and wacky uses you put this new release to! And finally, the lifeblood of Mirage is about sharing and publishing libraries that add new functionality to the framework, so do get involved and open-source your own efforts.

The Docker exploit and the security of containers

We normally only cover news and information directly related to Xen in this channel, but we thought it might be useful to briefly expand our scope a bit to mention the recent discussion about the Docker security exploit.

What’s the news?

Well to begin with, a few weeks ago Docker 1.0 was released, just in time for DockerCon.

Then last week, with timing that seems rather spiteful, someone released an exploit that allows a process running as root within a Docker container to break out of a Docker container.

I say it was a bit spiteful, because as the post-mortem on Docker’s site demonstrates, it exploits a vulnerability which was only present until Docker 0.11; it was fixed in Docker 0.12, which eventually became Docker 1.0.

Nonetheless, this kicked off a bit of a discussion about Docker, Linux containers, and security in several places, including Hacker News, the Register, seclists.org, and the OSv blog.

There is a lot of interesting discussion there. I recommend skimming through the Hacker News discussion in particular, if you have the time. But I think the comment on the Hacker News discussion by Solomon Hykes from Docker, puts it best:

As others already indicated this doesn’t work on 1.0. But it could have. Please remember that at this time, we don’t claim Docker out-of-the-box is suitable for containing untrusted programs with root privileges. So if you’re thinking “pfew, good thing we upgraded to 1.0 or we were toast”, you need to change your underlying configuration now. Add apparmor or selinux containment, map trust groups to separate machines, or ideally don’t grant root access to the application.

I’ve played around with Docker a little bit, and it seems like an excellent tool for packaging and deploying applications. Using containers to separate an application from the rest of the user-space of your distribution, exposing it only to the very forward-compatible Linux API is a really clever idea.

However, using containers for security isolation is not a good idea. In a blog last August, one of Docker’s engineers expressed optimism that containers would eventually catch up to virtual machines from a security standpoint. But in a presentation given in January, the same engineer said that the only way to have real isolation with Docker was to either run one Docker per host, or one Docker per VM. (Or, as Solomon Hykes says here, to use Dockers that trust each other in the same host or the same VM.)

We would concur with that assessment.

Security disclosure process discussion update

After concluding our poll about changes to the security discussion, we determined that “Pre-disclosure to software vendors and a wide set of users” was probably the best fit for the community. A set of concrete changes to the policy have now been discussed on xen-devel (here and here), and we seem to have converged on something everyone finds acceptable.

We are now presenting these changes for public review. The purpose of this review process is to allow feedback on the text which will be voted on, in accordance to the Xen.org governance procedure. Our plan is to leave this up for review until the third week in January. Any substantial updates will be mentioned on the blog and will extend the review time.

All feedback and discussion should happen in public on the xen-devel mailing list. If you have any suggestions for how to improve the proposal, please e-mail the list, and cc George Dunlap (george dot dunlap at citrix.com).

Read on for a summary of the updates, as well as links to the full text of the original and proposed new policies.
Continue reading

Xen.org Security Policy Update: Get Involved

Xen.org recently released a number of (related) security updates, XSA-7 through to -9. This was done by the Xen.org Security Team who are charged with following the Xen.org Security Problem Response Process.

As part of the process of releasing XSA-7..9 several short-comings (a few of which Ian Jackson has discussed already in Security vulnerabilities – the coordinated disclosure sausage mill) were found in the process.

In order to address these short-comings we have started a discussion on the xen-devel mailing list which describes the issues which we faced and proposes some potential options for updates. However this process is supposed to serve you, the Xen user community, and therefore your feedback and input is critical to ensuring that the policy meets the needs of the community.

So whether you are a small or large consumer of Xen you should feel free to have your say and to help formulate an updated policy which best serves the needs of the community. To take part in the discussion please send mail to xen-devel@lists.xen.org.

Security vulnerabilities – the coordinated disclosure sausage mill

Laws, like sausages, cease to inspire respect in proportion as we know how they are made. – John Godfrey Saxe, 1869.

Most open source projects, Xen.org included, do what is called “coordinated disclosure” of security problems. The idea is that we keep security bugs secret until people have had a chance to patch.

Mostly this process looks serene on the outside, but from the inside it can be very messy indeed. Particularly if, as happened recently with XSA-7 / CVE-2012-0217, large and powerful corporations apply pressure to keep the bug and the fix under wraps for months while their sclerotic update processes grind on.

Many of you will already know about this vulnerability, a bug in Intel’s implementation of the sysret instruction in AMD’s amd64 (aka x86_64) processor architecture. George Dunlap has already explained the technical details. This serious problem was discovered in the context of Xen and FreeBSD on the 9th of April. The fix was originally scheduled to go out on the 1st of May but in the end was not made available to all of you, the users, until the 12th of June.

There were some other problems too: we in the Xen.org security team made some key mistakes. We didn’t involve other organisations early enough, and the patches weren’t written carefully or reviewed closely enough.

So to try to make sure that things go better next time, the team have posted a formal request for discussion about how to improve the policy. This also contains, as an exercise in Free Software / Open Source transparency, a summary of what went on behind closed doors during the embargo period.

If you’ve ever wanted to see how the “coordinated disclosure” sausage is made, here’s a glimpse into that world. Warning: it may put you off. Hopefully it will put you off using the loaded term “responsible disclosure” for something which involves keeping the majority of deployed installations exposed for months to a bug which was first discovered in 2006.

Have your say!

So, following the request for discussion there is now a thread on the xen-devel mailing list to discuss and agree on improvements.

Continue reading