Author Archives: Liu Wei

Xen Hypervisor to be Rewritten

The hypervisor team has come to the conclusion that using the C programming language, which is 45 years old as of writing, is not a good idea for the long term success of the project.

C, without doubt, is ridden with quirks and undefined behaviours. Even the most experienced developers find this collection of powerful footguns difficult to use. We’re glad that the development of programming languages in the last decade has given us an abundance of better choices.

After a heated debate among committers, we’ve settled on picking two of the most popular languages on HackerNews to rewrite the Xen hypervisor project. Our winners are Rust and JavaScript.

Rust, although not old enough to drink, has attracted significant attention in recent years. The hypervisor maintainers have acquainted themselves with the ownership system, borrow checker, lifetimes and cargo build system. We will soon start rewriting the X86 exception handler entry point, which has been a major source of security bugs in the past, and looks like an easy starting point for the conversion to Rust.

JavaScript has been a corner stone of web development since early 2000. With the advancement of React Native and Electron, plus the exemplary success of Atom and Visual Studio Code editors, it now makes sense to start rebuilding the Xen hypervisor toolstack in JavaScript. We’re confident that Node.js would be of great help when it comes to performance. And we believe Node.js and the current libxenlight event model is a match made in heaven.

Due to the improved ergonomics of the two programming languages, we expect developer efficiency to be boosted by factor of 10. We’re also quite optimistic that we can tap into the large talent pool of Rust and JavaScript developers and get significant help from them. We expect the rewrite to be finished and released within the year – by April 2018.

For those who want a more solid, tried and true technology, we are open to the idea of toolstack middleware being written in PHP and frontend JavaScript. But since maintainers are too busy playing with their new shiny toys, those who want PHP middleware will have to step up and help.

Stay put and get ready to embrace the most secure and easy to use Xen hypervisor ever, on April 1st 2018!

Note that this article was an April fools joke and was entirely made up.

What’s New with Xen Project Hypervisor 4.8?

I’m pleased to announce the release of the Xen Project Hypervisor 4.8. As always, we focused on improving code quality, security hardening as well as enabling new features. One area of interest and particular focus is new feature support for ARM servers. Over the last few months, we’ve seen a surge of patches from various ARM vendors that have collaborated on a wide range of updates from new drivers to architecture to security.

We are also pleased to announce that Julien Grall will be the next release manager for Xen Project Hypervisor 4.9. Julien has been an active developer for the past few years, making significant code contributions to advance Xen on ARM. He is a software virtualization engineer at ARM and co-maintainer of Xen on ARM with Stefano Stabellini.

This release also marks the start of our first 6-month release cycle. Despite the shorter timeframe and putting more thorough security processes in place, we have maintained development momentum for Xen Project Hypervisor.

We’ve also worked with the Debian community to bring Xen Project Hypervisor 4.8 to the upcoming release (codename “Stretch”).

Here are the categories with updates to highlight in 4.8

  • Hypervisor General
  • Hypervisor x86
  • Hypervisor ARM
  • Toolstack
  • Xen Project Test Lab
  • Misc.

Hypervisor General

  • Credit2 scheduler is now supported: Compared to the default Credit scheduler, the Credit2 scheduler is more scalable and better at supporting latency sensitive workloads such as VDI, video and sound delivery, as well as unikernel applications. Credit2 is still based on a general purpose, weighted fair share, scheduling algorithm unlike some of the more specialized Xen Project schedulers such as RTDS and ARINC653.
  • Domain creation time optimisation: An optimisation to TLB flush is introduced to greatly reduce the number of flushes needed during domain creation. This has lead to the reduction of domain creation time for very large domains (with hundreds of gigabytes of RAM) from a few minutes to tens of seconds.
  • XSM policy is refactored and cleaned up: XSM policy files are refactored and cleaned up so that they are better organised and easier to understand. If configured, we can also now attach the in-tree default policy to Xen binary, so there is no need to load the default policy via boot loader.
  • Live Patching hook support: Live Patching is now able to look for the “hooks” section in the payload and execute code from there. This update gives the patch author more control in modifying data and code.

Hypervisor x86

  • CPUID faulting emulation: This makes CPUID fault in HVM userspace program without hardware support.
  • PVCLOCK_TSC_STABLE_BIT support: This greatly improves user space performance for time related syscalls.
  • Intel AVX-512 instructions support: These instructions offer higher performance for the most demanding computational tasks. They represent a significant leap to 512-bit SIMD support. This enables processing of twice the number of data elements that AVX/AVX2 can process with a single instruction and four times that of SSE.
  • PVH v2 DomU ABI is stabilised: The DomU guest ABI for PVH v2, without PCI passthrough support, is stabilised. Guest operating system developers can start porting OSes to this mode, which is simpler and gives them all the goodies that hardware and software provide.

Hypervisor ARM

  • Xen Project 4.8 ARM DomU ACPI support is now able to build ARM64 guests with ACPI support, such as Red Hat Enterprise Linux Server for ARM Development Preview (available via Partner Early Access Program). It can also run unmodified Xen on ARM.
  • Alternative patching support: This enables the hypervisor to apply workarounds for erratas affecting the processor and to apply optimizations specific to a CPU.
  • Live Patching initial support: Live Patching now supports both ARM32 and ARM64 platforms.
  • Support for Xilinx® Zynq® UltraScale+™ MPSoC: Xen Project Hypervisor 4.8 comes with support for the Xilinx Zynq UltraScale+ MPSoC making it much easier for Xilinx customers to integrate Xen into their solution.

Toolstack

  • Split out and re-license libacpi: The code inside hvmloader to construct guest ACPI tables is split out as a separate library libacpi, which is now shared across x86 and ARM. The code is re-licensed from GPL to LGPL.
  • HVM USB passthrough: It is now possible to passthrough USB devices to HVM guests with the help of QEMU.
  • Load BIOS via libxl: It is now possible to provide arbitrary BIOS binary to the guest making it easier to integrate and test Xen.
  • Libxl device handling framework: The device handling code inside libxl is reworked so that it is more extensible and easier to maintain.

Xen Project Test Lab

  • XTF is integrated into OSSTest: XTF is a micro-VM based test framework. It is now integrated into OSSTest and gates pushing patches to all supported Xen branches. This would help the project identify functional and security regressions more easily and quickly.

Misc.

  • Mini-OS ported to PVH v2: With the stabilization of PVH v2 DomU ABI, we are now confident to port mini-os to that mode. This would serve as an example to port guest OSes to PVH v2, as well as a foundation to more interesting micro-VM based work like building stub domains. The latter (stub domains) is a differentiator to other hypervisors, and could greatly enhance the security and scalability of Xen Project Hypervisor.
  • Mini-OS now supports ballooning up: Ideally, a service domain would need to dynamically adjust the memory it consumes, either voluntarily or via obeying command from hypervisor. This is an important feature to make Mini-OS based service domains more flexible in terms of memory consumption, which is one step towards that goal. Support for ballooning down Mini-OS is under development.

Summary

Despite the shorter release cycle, the community developed several major features, and found and fixed many more bugs. It is also rather impressive to see multiple vendors collaborate on the Xen Project Hypervisor to drive multiple projects forward. Major contributions for this release come from ARM, BitDefender, Bosch, Citrix, Freescale, Intel, Linaro, Oracle, Qualcomm, SUSE, Star Lab, the US National Security Agency, Xilinx, Zentific, and a number of universities and individuals.

Over the last year, contributors with strong security and embedded backgrounds have joined the Xen Project allowing us to  continue to focus on performance and flexibility without sacrificing security and reliability. Xen Project Hypervisor continues to move forward thanks to amazing efforts from companies developing products based on the hypervisor, such as XenServer 7 and Bitdefender Hypervisor Introspection, and novel new developments with Live Patching and Virtual Machine Introspection.

In this release, we took a security-first approach and spent a lot of energy to improve code quality and harden security. This inevitably slowed down the acceptance of new features a bit, but not enough to reach meaningful balance between mature security practice and innovation.

On behalf of the Xen Project Hypervisor team, I would like to thank everyone for their contributions (either in the form of patches, bug reports or packaging efforts) to the Xen Project. Please check our acknowledgement page, which recognizes all those who helped make this release happen.

The source can be located in the http://xenbits.xen.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.8 tree (tag RELEASE-4.8.0) or can be downloaded as tarball from our website. More information can be found at

Please Welcome Our New Release Manager

Dear community members,

I’m pleased to announce that Julien Grall <julien.grall@arm.com> will be the Release Manager for the next Xen release.

The appointment was voted by the Committers and the vote passed.

Julien has done excellent jobs in many aspects. He has been an active  developer for the past few years and contributed a lot of code for Xen on ARM. He has been doing a good job in co-maintaining Xen on ARM with Stefano Stabellini. Particularly in 4.8 release, he showed his ability to make balanced decisions and influence other contributors to move various projects forward. He also expressed desire to work with greater Xen community and make bigger impact.

All in all, we believe Julien will do a good job in managing the next release. Thanks Julien for stepping up.

Regards,
Wei Liu (on behalf of the Xen Project Hypervisor team)

Xen Project 4.7 Planning Opens

With Xen 4.6 released in October, we are already one month into the new cycle. Which means it is time to start planning for the next release. You may remember that one of the goals of the 4.6 release planning was to create smoother developer experience and to release Xen 4.6 on time. Both goals were achieved, so it was time to think where to go from here. Thus, the Xen community underwent a thorough discussion on how to manage future releases from xen-unstable and its impact on stable releases. The takeaway message of those lengthy threads is that we should continue to work on making the release cycle shorter and more predictable.

As such, the timeline for 4.7 is:

  • Development starts: October 13, 2015
  • Last posting date: March 18, 2016
  • Hard code freeze: April 1, 2016
  • Release date: June 3, 2016

After the 4.7 release, we will start to release Xen every 6 months: at the beginning of June and December. A regular 6 monthly release schedule has worked well for Ubuntu, OpenStack and many other projects. The idea behind it is a simple one: set a hard date and modify your goals to match that timeline. Which is also, why we dropped feature freeze exceptions, which create overheads and introduce unnecessary risk and debate. In addition, the new fixed release schedule will help open source projects and commercial vendors who consume Xen to plan their own releases better. And it allows us to set a schedule that ensures that every single release cycle is only affected by a single holiday period and that we have a Xen Project developer event (be it a Hackathon or Xen Project Developer Summit) during each release cycle. The stable release scheme is unchanged: 18 months full support, plus 18 months security fixes afterwards.

For more information, check out the slides that explain our release process and how it is changing for Xen 4.7 and beyond. To follow the roadmap in the coming months, be sure to check the Xen 4.7 Roadmap page on our wiki. Get involved on xen-devel@ and happy hacking!

For more updates, follow @XenProject.org on Twitter.

Best Quality and Quantity of Contributions in the New Xen Project 4.6 Release

I’m pleased to announce the release of Xen Project Hypervisor 4.6. This release focused on improving code quality, security hardening, enablement of security appliances, and release cycle predictability — this is the most punctual release we have ever had. We had a significant amount of contributions from cloud providers, software vendors, hardware vendors, academic researchers and individuals to help with this release. We continue to strive to make Xen Project Hypervisor the most secure open source hypervisor to match the security challenges in cloud computing, and for embedded and IoT use-cases. We are also continuing to improve upon the performance and scalability for our users, and aim to continuously bring many new features to our users in a timely manor.

Despite an increase of new features compared to previous releases, the Xen Project Hypervisor codebase has only increased by 6KLOC compared to Xen 4.5. In addition, we were able to increase the number of changesets that we integrated into Xen from 178/month (1812 in total) for Xen 4.5 to 259/month (2247 in total). In addition, the quality of Xen 4.6 was higher than in the past, enabling the CentOS 7 Virtualization SIG and XenServer to include Xen into their upcoming releases.

To make it easier to understand the major changes during this release cycle I have grouped the major updates into several categories:

  • Hypervisor
  • Toolstack
  • Xen Project Test Lab
  • Linux, FreeBSD and other OSes that utilise the new features
  • Greater Ecosystem

General Hypervisor Updates

  • The memory event subsystem has been reworked and extended to a new VM event subsystem. The new VM event subsystems supports both the ARM and x86 architectures. It can be used to intercept all sorts of VM events, such as memory access, register access and more. This enables security applications such as zero-footprint guest introspection, host-wide monitoring and many others. Have a look at Tamas’s presentations and Steve’s presentations on this topic to get more insights.
  • The Xen Security Modules (XSM) now have a default policy that is regularly tested in the Xen Project Test Lab to make sure it is not broken by mistake. This will enable us to switch on XSM by default in the future.
  • vTPM 2.0 support has been contributed by Intel and BitDefender [ 1 ]. To learn more about how to use vTPM and how it can make your host more secure, go to our wiki.
  • Grant table scalability has been improvement significantly by using finer-grained locks in grant tables. In some scenarios aggregate intrahost network throughput has been shown to improve by 100%. Other I/O drivers in Xen should potentially show significant performance improvements as well.
  • We introduced ticket lock to improve fairness, which provides better support of massive workloads from up to hundreds or thousands of VMs on a single host.
  • The unused SEDF scheduler has been removed from the hypervisor and toolstack. The Xen Project is committed to actively remove unused code to keep the code base small and to minimize security risks.
  • We removed Mini-OS from the Xen code base into its own tree. Mini-OS started as a demonstration OS, but received significant contributions in recent years (e.g. it is used by many Unikernels). We decide to treat it as a separately maintained independent project with it’s own mailing list and code tree to make it easier to consume. We hope this will help unikernel communities to more easily consume and contribute to Mini-OS, while reducing the Xen Project Hypervisor footprint.

x86-specific Hypervisor Updates

  • The Intel alternate P2M framework is a new capability for VM Introspection, Security and Privacy in Xen that gives Xen the ability to host up to 10 alternate guest to physical memory domains mappings for a specific guest-domain. It is one of the key technologies to enable zero-footprint VM introspection. It can also help Xen to implement faster NFV applications.
  • Intel Page Modification Logging Technology offloads the page dirty logging duty to hardware. Microbenchmark shows about 7% improvement in SPECJbb and should be particularly beneficial for Live Migration.
  • Intel Cache Allocation Technology allows system administrators to assign more L3 cache capacity to individual VMs, resulting in lower latency and higher performance for high-priority workloads such as NFV, real-time and video-on-demand applications.
  • Intel Memory Bandwidth Monitoring allows system administrators to identify memory bandwidth saturation on a Xen host that may be caused by several memory-intensive VMs running on the same host. Taking corrective actions, such as migrating VMs to a different Xen host, increases scalability and performance in the data center.
  • Intel Reserve Memory Region reporting provides a mechanism to report and reserve memory regions for legacy devices to allow for safe device passthrough.
  • Virtual Performance Monitoring Unit support makes it possible to profile the Xen Project Hypervisor with the Linux perf tool. Note that some work still needs to be completed within Linux to make perf fully functional.
  • Virtual NUMA for HVM guest is a continuation of the NUMA work performed in Xen 4.5 and previous releases. In this release, we exposed the functionality through the XL toolstack and added firmware changes to make the feature fully functional.

ARM-specific Hypervisor Updates

  • The supported number of VCPUs has been increased from 8 to 128 VCPUs on ARM64 platforms.
  • Passthrough for non-PCI devices allows users to passthrough devices via partial device trees. Full support for PCI device passthrough is currently being worked on.
  • ARM GICv2 on GICv3 support.
  • 32 bit userspace in 64 bit guest support.
  • OVMF for ARM contributed by Linaro.
  • 64K page ARM guest support.
  • Support for the following new Hardware Platforms has been added: Renesas R-Car Gen2, Thunder X, Huawei hip04-d04 and Xilinx ZynqMP SoC.

Toolstack Updates

  • Live Migration support in libxc / libxl and has been replaced with a completely new implementation (Migration v2). The new version respects different layers in the Xen Software stack and has been designed to be more robust and extensible to better support next-generation infrastructures and work planned in subsequent hypervisor releases.
  • Remus – our High Availability solution – has been reworked and is now based on Migration v2.
  • Libxl asynchronous operations can now be cancelled. This allows libxl users to cancel long-running asynchronous operations and benefits tool stacks such as libvirt and is beneficial for integration with cloud orchestration stacks.
  • Improved SPICE/QXL support.
  • AHCI disk controller support.
  • A new host I/O topology query interface gives upper layer in the software stack the ability to identify the I/O topology of underlying hardware platform.
  • Addition of Xenalyze, which is a tool for analyzing Hypervisor trace buffers and can be used for debugging and optimization, has been added to the Xen Project codebase as a maintained feature.

Xen Project Test Lab Updates

During the Xen 4.6 release cycle, the Xen Project created an Advisory Board funded Continuous Integration Test Lab. It currently has 24 hosts and is going to expanded in the future. This has led to significant improvements in Xen code quality and has allowed the project to expand automated test coverage. The number of test cases doubled during the 4.6 cycle. Some interesting new test cases that have been added are:

  • XSM
  • Stub Domain
  • VM migration using libvirt between two hosts is now tested.
  • Live Migration between hosts of different Xen versions is now tested and will help identify any breakage in our migration code or specification.
  • Test with different disk formats such as QCOW2, VHD and raw format.

More test cases are in the pipeline, including test case for OpenStack’s devstack, performance and scalability tests, FreeBSD Dom0 etc.

Linux, FreeBSD and other OSes

During the Xen 4.6 release cycle, we made significant improvements to major operating systems we rely on to improve interoperability. Some highlights on Linux kernel development spanning from Linux 3.18 to 4.3 were:

  • Xen blkfront multiqueue and multipage ring support.
  • Xen SCSI frontend and backend support.
  • VPMU kernel support.
  • Performance improvement in mmap call.
  • P2M in PV guest can address 512GB or more.

For FreeBSD there were these improvements:

  • Experimental PVH Dom0/DomU support.
  • Removal of classic i386 PV port by FreeBSD developer John Baldwin.
  • Blkfront indirect descriptor support by FreeBSD developer Colin Percival.
  • Removal of broken FreeBSD specific blkfront/back extensions.
  • ARM32 and ARM64 guest support are underway.

Greater Ecosystem

Summary

With dozens of major improvements, many more bug fixes and small improvements, efforts in other projects as well as a greater ecosystem, Xen 4.6 reflects a thriving community around the Xen Project Hypervisor. We are extremely proud of achieving the highest quality of the release while increasing development velocity. In particular, our latest security related features enable Xen to compete in the security appliance market and help answer some of the difficult questions regarding security in the cloud era.

We set out at the beginning of this release cycle to foster greater collaboration among vendors, individual developers, upstream maintainers, other projects and distributions. During this release cycle we continued to see an increasing influx of patches and newcomers. As the release manager, I would like to thank everyone for their contributions (either in the form of patches, bug reports or packaging efforts) to Xen. This release wouldn’t have happened without contributions from so many people around the world. Please check out our 4.6 contributor acknowledgement page.

The source can be located in the xen.git tree (tag RELEASE-4.6.0) or can be downloaded tarball from our website. More information can be found at


[ 1 ] Note that when this article was published, the contribution was mistakenly attributed to the US National Security Agency, instead of BitDefender.

Xen Project 4.6 RC2 Test Day is September 1, 2015

Join 4.6 Release Candidate Testing on September 1, 2015

39833137_mAlthough the Xen Project performs automated testing through the project’s Test Lab, we also depend on manual testing of release candidates by our users. Our Test Days help insure that upcoming releases are ready for production. It is particularly important that our users test out the upcoming release in their own environment. In addition, functional testing of features (in particular those which can’t be automated), stress-testing, edge case testing and performance testing are important for a new release.

Xen 4.6 Release Candidate Testing

A few weeks ago, Xen 4.6 went into code freeze and Xen 4.6 RC2 is now ready for testing. With this in mind the Test Day for Xen 4.6 RC2 has been set for next Tuesday, September 1, 2015.

Subsequent Test Days are expected to be scheduled roughly ever other week until Xen 4.6 is ready for release.

Test Day Information

General Information about Test Days can be found here:

Join us on Tuesday in #xentest on Freenode IRC!
Test a Release Candidate! Help others, get help! And have fun!
If you can’t make Tuesday, remember that Test and Issue Reports are welcome any time.