Xen Project 4.7 Planning Opens

With Xen 4.6 released in October, we are already one month into the new cycle. Which means it is time to start planning for the next release. You may remember that one of the goals of the 4.6 release planning was to create smoother developer experience and to release Xen 4.6 on time. Both goals were achieved, so it was time to think where to go from here. Thus, the Xen community underwent a thorough discussion on how to manage future releases from xen-unstable and its impact on stable releases. The takeaway message of those lengthy threads is that we should continue to work on making the release cycle shorter and more predictable.

As such, the timeline for 4.7 is:

  • Development starts: October 13, 2015
  • Last posting date: March 18, 2016
  • Hard code freeze: April 1, 2016
  • Release date: June 3, 2016

After the 4.7 release, we will start to release Xen every 6 months: at the beginning of June and December. A regular 6 monthly release schedule has worked well for Ubuntu, OpenStack and many other projects. The idea behind it is a simple one: set a hard date and modify your goals to match that timeline. Which is also, why we dropped feature freeze exceptions, which create overheads and introduce unnecessary risk and debate. In addition, the new fixed release schedule will help open source projects and commercial vendors who consume Xen to plan their own releases better. And it allows us to set a schedule that ensures that every single release cycle is only affected by a single holiday period and that we have a Xen Project developer event (be it a Hackathon or Xen Project Developer Summit) during each release cycle. The stable release scheme is unchanged: 18 months full support, plus 18 months security fixes afterwards.

For more information, check out the slides that explain our release process and how it is changing for Xen 4.7 and beyond. To follow the roadmap in the coming months, be sure to check the Xen 4.7 Roadmap page on our wiki. Get involved on xen-devel@ and happy hacking!

For more updates, follow @XenProject.org on Twitter.

Xen Project 4.5.2 Maintenance Release Available

I am pleased to announce the release of Xen 4.5.2. Xen Project Maintenance releases are released roughly every 4 months, in line with our Maintenance Release Policy. We recommend that all users of the 4.5 stable series update to this point release.

Xen 4.5.2 is available immediately from its git repository:

    xenbits.xenproject.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.5
    (tag RELEASE-4.5.2)

or from the Xen Project download page at www.xenproject.org/downloads/xen-archives/xen-45-series/xen-452.html.

This release contains many bug fixes and improvements. For a complete list of changes in this release, please check lists of changes on the download page.

Security vs Features

We’ve just released a rather interesting batch of Xen security advisories. This has given rise in some quarters to grumbling around Xen not taking security seriously.

I have a longstanding interest in computer security. Nowadays I am a member of the Xen Project Security Team (the team behind security@xenproject, which drafts the advisories and coordinates the response). But I’m going to put forward my personal opinions.

Of course Invisible Things are completely right that security isn’t taken seriously enough. The general state of computer security in almost all systems is very poor. The reason for this is quite simple: we all put up with it. We, collectively, choose convenience and functionality: both when we decide which software to run for ourselves, and when we decide what contributions to make to the projects we care about. For almost all software there is much stronger pressure (from all sides) to add features, than to improve security.

That’s not to say that many of us working on Xen aren’t working to improve matters. The first part of improving anything is to know what the real situation is. Unlike almost all corporations, and even most Free Software projects, the Xen Project properly discloses, via an advisory, every vulnerability discovered in supported configurations. We also often publish advisories about vulnerabilities in other relevant projects, such as Linux and QEMU.

Security bugs are bugs, and over the last few years the Xen Project’s code review process has become a lot more rigorous. As a result, the quality of code being newly introduced into Xen has improved a lot.

For researchers developing new analysis techniques, Xen is a prime target. A significant proportion of the reports to security@xenproject are the result of applying new scanning techniques to our codebase. So our existing code is being audited, with a focus on the areas and techniques likely to discover the most troublesome bugs.

As I say, the Xen Project is very transparent about disclosing security issues; much more so than other projects. This difference in approach to disclosure makes it difficult to compare the security bug density of competing systems. When I worked for a security hardware vendor I was constantly under pressure to explain why we needed to do a formal advisory for our bugs. That is what security-conscious users expect, but our competitors’ salesfolk would point to our advisories and say that our products were full of bugs. Their products had no publicly disclosed security bugs, so they would tell naive customers that their products had no bugs.

I do think Xen probably has fewer critical security bugs than other hypervisors (whether Free or proprietary). It’s the best available platform for building high security systems. The Xen Project’s transparency is very good for Xen’s users. But that doesn’t mean Xen is good enough.

Ultimately, of course, a Free Software project like Xen is what the whole community makes it. In the project as a whole we get a lot more submissions of new functionality than we get submissions aimed at improving the security.

So personally, I very much welcome the contributions made by security-focused contributors – even if that includes criticism.

Best Quality and Quantity of Contributions in the New Xen Project 4.6 Release

I’m pleased to announce the release of Xen Project Hypervisor 4.6. This release focused on improving code quality, security hardening, enablement of security appliances, and release cycle predictability — this is the most punctual release we have ever had. We had a significant amount of contributions from cloud providers, software vendors, hardware vendors, academic researchers and individuals to help with this release. We continue to strive to make Xen Project Hypervisor the most secure open source hypervisor to match the security challenges in cloud computing, and for embedded and IoT use-cases. We are also continuing to improve upon the performance and scalability for our users, and aim to continuously bring many new features to our users in a timely manor.

Despite an increase of new features compared to previous releases, the Xen Project Hypervisor codebase has only increased by 6KLOC compared to Xen 4.5. In addition, we were able to increase the number of changesets that we integrated into Xen from 178/month (1812 in total) for Xen 4.5 to 259/month (2247 in total). In addition, the quality of Xen 4.6 was higher than in the past, enabling the CentOS 7 Virtualization SIG and XenServer to include Xen into their upcoming releases.

To make it easier to understand the major changes during this release cycle I have grouped the major updates into several categories:

  • Hypervisor
  • Toolstack
  • Xen Project Test Lab
  • Linux, FreeBSD and other OSes that utilise the new features
  • Greater Ecosystem

General Hypervisor Updates

  • The memory event subsystem has been reworked and extended to a new VM event subsystem. The new VM event subsystems supports both the ARM and x86 architectures. It can be used to intercept all sorts of VM events, such as memory access, register access and more. This enables security applications such as zero-footprint guest introspection, host-wide monitoring and many others. Have a look at Tamas’s presentations and Steve’s presentations on this topic to get more insights.
  • The Xen Security Modules (XSM) now have a default policy that is regularly tested in the Xen Project Test Lab to make sure it is not broken by mistake. This will enable us to switch on XSM by default in the future.
  • vTPM 2.0 support has been contributed by Intel and BitDefender [ 1 ]. To learn more about how to use vTPM and how it can make your host more secure, go to our wiki.
  • Grant table scalability has been improvement significantly by using finer-grained locks in grant tables. In some scenarios aggregate intrahost network throughput has been shown to improve by 100%. Other I/O drivers in Xen should potentially show significant performance improvements as well.
  • We introduced ticket lock to improve fairness, which provides better support of massive workloads from up to hundreds or thousands of VMs on a single host.
  • The unused SEDF scheduler has been removed from the hypervisor and toolstack. The Xen Project is committed to actively remove unused code to keep the code base small and to minimize security risks.
  • We removed Mini-OS from the Xen code base into its own tree. Mini-OS started as a demonstration OS, but received significant contributions in recent years (e.g. it is used by many Unikernels). We decide to treat it as a separately maintained independent project with it’s own mailing list and code tree to make it easier to consume. We hope this will help unikernel communities to more easily consume and contribute to Mini-OS, while reducing the Xen Project Hypervisor footprint.

x86-specific Hypervisor Updates

  • The Intel alternate P2M framework is a new capability for VM Introspection, Security and Privacy in Xen that gives Xen the ability to host up to 10 alternate guest to physical memory domains mappings for a specific guest-domain. It is one of the key technologies to enable zero-footprint VM introspection. It can also help Xen to implement faster NFV applications.
  • Intel Page Modification Logging Technology offloads the page dirty logging duty to hardware. Microbenchmark shows about 7% improvement in SPECJbb and should be particularly beneficial for Live Migration.
  • Intel Cache Allocation Technology allows system administrators to assign more L3 cache capacity to individual VMs, resulting in lower latency and higher performance for high-priority workloads such as NFV, real-time and video-on-demand applications.
  • Intel Memory Bandwidth Monitoring allows system administrators to identify memory bandwidth saturation on a Xen host that may be caused by several memory-intensive VMs running on the same host. Taking corrective actions, such as migrating VMs to a different Xen host, increases scalability and performance in the data center.
  • Intel Reserve Memory Region reporting provides a mechanism to report and reserve memory regions for legacy devices to allow for safe device passthrough.
  • Virtual Performance Monitoring Unit support makes it possible to profile the Xen Project Hypervisor with the Linux perf tool. Note that some work still needs to be completed within Linux to make perf fully functional.
  • Virtual NUMA for HVM guest is a continuation of the NUMA work performed in Xen 4.5 and previous releases. In this release, we exposed the functionality through the XL toolstack and added firmware changes to make the feature fully functional.

ARM-specific Hypervisor Updates

  • The supported number of VCPUs has been increased from 8 to 128 VCPUs on ARM64 platforms.
  • Passthrough for non-PCI devices allows users to passthrough devices via partial device trees. Full support for PCI device passthrough is currently being worked on.
  • ARM GICv2 on GICv3 support.
  • 32 bit userspace in 64 bit guest support.
  • OVMF for ARM contributed by Linaro.
  • 64K page ARM guest support.
  • Support for the following new Hardware Platforms has been added: Renesas R-Car Gen2, Thunder X, Huawei hip04-d04 and Xilinx ZynqMP SoC.

Toolstack Updates

  • Live Migration support in libxc / libxl and has been replaced with a completely new implementation (Migration v2). The new version respects different layers in the Xen Software stack and has been designed to be more robust and extensible to better support next-generation infrastructures and work planned in subsequent hypervisor releases.
  • Remus – our High Availability solution – has been reworked and is now based on Migration v2.
  • Libxl asynchronous operations can now be cancelled. This allows libxl users to cancel long-running asynchronous operations and benefits tool stacks such as libvirt and is beneficial for integration with cloud orchestration stacks.
  • Improved SPICE/QXL support.
  • AHCI disk controller support.
  • A new host I/O topology query interface gives upper layer in the software stack the ability to identify the I/O topology of underlying hardware platform.
  • Addition of Xenalyze, which is a tool for analyzing Hypervisor trace buffers and can be used for debugging and optimization, has been added to the Xen Project codebase as a maintained feature.

Xen Project Test Lab Updates

During the Xen 4.6 release cycle, the Xen Project created an Advisory Board funded Continuous Integration Test Lab. It currently has 24 hosts and is going to expanded in the future. This has led to significant improvements in Xen code quality and has allowed the project to expand automated test coverage. The number of test cases doubled during the 4.6 cycle. Some interesting new test cases that have been added are:

  • XSM
  • Stub Domain
  • VM migration using libvirt between two hosts is now tested.
  • Live Migration between hosts of different Xen versions is now tested and will help identify any breakage in our migration code or specification.
  • Test with different disk formats such as QCOW2, VHD and raw format.

More test cases are in the pipeline, including test case for OpenStack’s devstack, performance and scalability tests, FreeBSD Dom0 etc.

Linux, FreeBSD and other OSes

During the Xen 4.6 release cycle, we made significant improvements to major operating systems we rely on to improve interoperability. Some highlights on Linux kernel development spanning from Linux 3.18 to 4.3 were:

  • Xen blkfront multiqueue and multipage ring support.
  • Xen SCSI frontend and backend support.
  • VPMU kernel support.
  • Performance improvement in mmap call.
  • P2M in PV guest can address 512GB or more.

For FreeBSD there were these improvements:

  • Experimental PVH Dom0/DomU support.
  • Removal of classic i386 PV port by FreeBSD developer John Baldwin.
  • Blkfront indirect descriptor support by FreeBSD developer Colin Percival.
  • Removal of broken FreeBSD specific blkfront/back extensions.
  • ARM32 and ARM64 guest support are underway.

Greater Ecosystem

Summary

With dozens of major improvements, many more bug fixes and small improvements, efforts in other projects as well as a greater ecosystem, Xen 4.6 reflects a thriving community around the Xen Project Hypervisor. We are extremely proud of achieving the highest quality of the release while increasing development velocity. In particular, our latest security related features enable Xen to compete in the security appliance market and help answer some of the difficult questions regarding security in the cloud era.

We set out at the beginning of this release cycle to foster greater collaboration among vendors, individual developers, upstream maintainers, other projects and distributions. During this release cycle we continued to see an increasing influx of patches and newcomers. As the release manager, I would like to thank everyone for their contributions (either in the form of patches, bug reports or packaging efforts) to Xen. This release wouldn’t have happened without contributions from so many people around the world. Please check out our 4.6 contributor acknowledgement page.

The source can be located in the xen.git tree (tag RELEASE-4.6.0) or can be downloaded tarball from our website. More information can be found at


[ 1 ] Note that when this article was published, the contribution was mistakenly attributed to the US National Security Agency, instead of BitDefender.

Xen now available in CentOS 7 for ARM64 servers

A little more than a week ago at Linaro Connect SFO15 in Burlingame Jim Perrin of the CentOS project publicly announced the availability of the Xen hypervisor in CentOS 7 for ARM64 (also known as aarch64). Jim and I have been working closely with George Dunlap, maintainer of Xen in CentOS for the x86 architecture, to produce high quality Xen binaries for 64-bit ARM servers. As a result you can setup an ARM64 virtualization host with just a couple of yum commands.

CentOS 7 aarch64 is available here. Installation is trivial: download the live image, try it out, and write it to disk if you like it. You can easily extend the root partition and filesystem to match the size of your disk.

Once you have CentOS 7 up and running on your ARM64 server, you can install Xen and Libvirt with the following commands:

yum install centos-release-xen
yum update
yum install xen libvirt

If you are using AppliedMicro X-Gene, you need to add a Xen command line option to specify which serial to use. This is due to the firmware missing one piece of information. We are working with AppliedMicro to fix the issue as soon as possible. In the meantime you can edit /etc/default/grub and add the following to GRUB_CMDLINE_XEN_DEFAULT:

dtuart=/soc/serial@1c020000

recreate the grub config file:

grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

Reboot and you’ll have Xen and Libvirt ready to use! Simple, right? :-)

If you want to try running a CentOS guest, just download the CentOS 7 live image, unpack it, and write a basic VM config file, using the Dom0 kernel and initramfs. For example:

kernel="/boot/vmlinuz-4.2.0-0.1.centos.el7.aarch64"
ramdisk="/boot/initramfs-4.2.0-0.1.centos.el7.aarch64.img"
memory=512
root="/dev/xvda4"
disk=[ "file:/path/to/CentOS-7-aarch64-rolling.img,xvda,w" ]
name = "centos7"
vcpus = 1
extra="console=hvc0"

Use xl to create the guest and connect to its console:

xl create -c config

Rinse and repeat as many times as you like, and you’ll have many little CentOS virtual machines keeping you company.

Xen 4.6 will be released shortly and you can count on us updating the Xen rpm in CentOS 7 shortly after. You’ll be able to install the latest and greatest Xen hypervisor release for ARM64 with a simple yum install.

At Linaro Connect I went further by showing ready to use OpenStack packages for CentOS 7 aarch64. Thanks to Anthony Perard, who produced those rpms, setting up Nova with Xen on ARM64 is just a matter of installing the packages and starting Nova services. Jim promised to have the OpenStack rpms online at centos.org in a couple of weeks. Stay tuned!

Xen Project Participates in Round 11 of Outreachy

This is a quick reminder that the Xen Project is again participating in Outreachy (Round 11). Please check the round 11 page for more information about the December 2015 to March 2015 round of interships.

Outreach Program for Women has been helping women (cis and trans), trans men, and genderqueer people get involved in free and open source software worldwide. Note that the program has been extended and now also targets people from more groups underrepresented in FOSS: specifically the program is open to residents and nationals of the United States of any gender who are Black/African American, Hispanic/Latin, American Indian, Alaska Native, Native Hawaiian, or Pacific Islander.

Round 11 of Outreachy

outreachy-poster-2015-December-March
The application deadline for interns is November 2, 2015. For a list of projects for interns and more information on how to apply, check our Xen Project Outreachy portal.

We have many different projects in many different areas! Check out the following table, which lists projects covering

  • Hypervisor Development (requiring Linux/BSD, C, scripting skills – there is also a Windows related project)
  • Mirage OS Development (requiring Linux/BSD, OCaml or Functional programming skills)
  • A Xen Code Review Dashboard project (requiring SQL, Javascript, HTML5 skills)

Learn about the Experience of past Applicants

At the 2014 Xen Project Developer Summit, we ran a panel discussion that included OPW interns, GSoC students as well as mentors.


You may also want to read Women interns rocking open source at Xen Project.

Looking forward to hear from you!