Author Archives: Julien Grall

About Julien Grall

Julien Grall is a Senior Software Engineer at ARM, working on open source virtualization. He has been working on Xen since 2012, initially focusing on Xen x86 and then on support for ARM architecture. He is currently a maintainer of Xen ARM.

What’s New in the Xen Project Hypervisor 4.10

I am pleased to announce the release of the Xen Project Hypervisor 4.10. As always, we focused on improving code quality, security hardening as well as enabling new features.

The Xen Project Hypervisor 4.10 continues to take a security-first approach with improved architecture and more centralized documentation. The release is equipped with the latest hardware updates from Arm and a more intuitive user interface.

We are also pleased to announce that Jürgen Groß will be the next release manager for Xen Project Hypervisor 4.11. Jürgen has been an active developer for the past few years, making significant code contributions to advance Xen support in Linux. He is a virtualization kernel developer at Suse and maintainer of Xen subsystem in Linux as well as parvirtualization.

We grouped updates to the Xen Project Hypervisor using the following categories:

  • Hypervisor general
  • Hypervisor Arm
  • Hypervisor x86
  • Toolstack
  • Misc

Hypervisor General

Credit 2 scheduler improvements: Soft-affinity support for the Credit 2 scheduler was added to allow those using the Xen Project in the cloud and server space to specify a preference for running a VM on a specific CPU. This enables NUMA aware scheduling for the Credit 2 scheduler. In addition we added cap support allowing users to set a the maximum amount of CPU a VM will be able to consume, even if the host system has idle CPU cycles.

Null scheduler improvements: The recent updates to the “null” scheduler guarantee near zero scheduling overhead, significantly lower latency, and more predictable performance. Added tracing support enables users to optimise workloads and introduced soft-affinity. Soft affinity adds a flexible way to express placement preference of vcpus on processors, which improves cache and memory performance when configured appropriately.

Virtual Machine Introspection improvements: Performance improvements have been made to VMI. A software page table walker was added to VMI on ARM, which lays the groundwork to alt2pm for ARM CPUs. For more information on alt2pm is available here.

PV Calls Drivers in Linux: In Xen Project 4.9, the Xen Project introduced the PV Calls ABI, which allows forwarding POSIX requests across guests. This enables a new networking model that is a natural fit for cloud-native apps. The PV Calls backend driver was added to Linux 4.14.

Better User Experience through the Xen Project User Interface

The Xen Project community also made significant changes to the hypervisor’s user interface. It is now possible to modify certain boot parameters without the need to reboot Xen. Guest types are now selected using the type option in the configuration file, where users can select a PV, PVH or HVM guest. The builder option is being depreciated in favor of the type option,  the PVH option has been removed and a set of PVH specific options have been added.

These changes allow the Xen Project to retain backward compatibility on new hardware without old PV code, providing the same functionality with a much smaller codebase. Additional user interface improvements are detailed in our blog post.

Hypervisor Arm

Support for Latest System-on-chip (SoC) Technology: The Xen Project now supports SoCs based on the 64-bit Armv8-A architecture from Qualcomm Centriq 2400 and Cavium ThunderX.

SBSA UART Emulation for Arm® CPUs: Implementation of SBSA UART emulation support in the in the Xen Project Hypervisor makes it accessible through the command line tools. This enables the guest OS to access the console when no PV console driver is present. In addition, the SBSA UART emulation is also required to be compliant with the VM System specification.

ITS support for ARM CPUs: Xen Project 4.10 adds support for ARM’s Interrupt Translation Service (ITS), which accompanies the GICv3 interrupt controller such as the ARM CoreLink GIC-500. ITS support allows the Xen Project Hypervisor to harnesses all of the benefits of the GICv3 architecture, improving interrupt efficiency and allowing for greater virtualization on-chip for both those using the Xen Project for the server and embedded space. ITS support is essential to virtualize systems with large amounts of interrupts. In addition ITS increases isolation of virtual machines by providing interrupt remapping, enabling safe PCI passthrough on ARM..

GRUB2 on 64-bit Armv8-A architecture: The GRUB community merged support to boot Xen on 64-bit Arm-based CPU platforms. GRUB2 support for Armv8-A improves the user experience when installing Xen via distribution package on UEFI platform.

Hypervisor x86

Rearchitecture Creates Smaller Attack Surface and Cleaner Code

Since the introduction of Xen Project Hypervisor 4.8, the project has overhauled the x86 core of its technology. The intention is to create a cleaner architecture, less code and a smaller computing base for security and performance. As part of this re-architecture, Xen Project 4.10 supports PVHv2 DomU. PVHv2 guests have a smaller TCB and attack surface compared to PV and HVM guests.

In Xen Project Hypervisor 4.9, the interface between Xen Project software and QEMU was completely reworked and consolidated via DMOP. For the Xen Project Hypervisor 4.10, the Xen Project community built on DMOP and added a Technology Preview for dm_restrict to constrain what device models, such as QEMU, can do after startup. This feature limits the impact of security vulnerabilities in QEMU. Any previous QEMU vulnerabilities that could normally be used for escalation privileges to the host cannot escape the sandbox.

This work significantly reduces potential security vulnerabilities in the Xen Project software stack.

L2 CAT for Intel CPUs: In Xen 4.10 we added support for Intel’s L2 Cache Allocation Technology(CAT) — available on certain models of (Micro) Server platforms. Xen L2 CAT support provides Xen users a mechanism to partition or share the L2 Cache among virtual machines, if such technology is present on the hardware Xen runs. This allows users to make better use of the shared L2 cache depending on the VM characteristic (e.g. priority).

Local Machine-Check Exception(LMCE) for Intel CPUs: Xen 4.10 provides LMCE support for HVM guests. A LMCE, if the affected vCPU is known, will be injected to related vCPU, otherwise, the LMCE will be broadcasted to all vCPUs running on the host. This allows for more efficient passing of MCE from hypervisor to virtual machines for further handling.

User Mode Instruction Prevention(UMIP) for Intel CPUs: User-Mode Instruction Prevention (UMIP) is a security feature present in new Intel Processors. If enabled, it prevents the execution of certain instructions if the Current Privilege Level (CPL) is greater than 0. Xen 4.10 exposes UMIP to virtual machines to take advantage of this feature.

Misc.

Improved Support Documentation

In Xen Project 4.10, a machine-readable file (support.md) was added to describe support related information in a single document. It defines support status and whether features are security supported and to which degree. For example, a feature may be security supported on x86, but not on Arm.

This file will be back-ported to older Xen releases and will be used to generate support information for Xen Project releases and will be published on xenbits.xen.org/docs/. This effort will both allow users to better understand how they are impacted by security issues, and centralizing security support related information is a pre-condition to become a CVE Numbering authority.

Summary

Despite the shorter release cycle, the community developed several major features, and found and fixed many more bugs. It is also rather impressive to see multiple vendors collaborate on the Xen Project Hypervisor to drive multiple projects forward. Contributions for this release of the Xen Project came from Amazon Web Services, AMD, Aporeto, Arm, BAE Systems, BitDefender, Cavium, Citrix, EPAM, GlobalLogic, Greenhost, Huawei Technologies, Intel, Invisible Things Lab, Linaro, Nokia, Oracle, Red Hat, Suse, US National Security Agency, and a number of universities and individuals.

As in Xen 4.9, we took a security-first approach for Xen 4.10 and spent a lot of energy to improve code quality and harden security. This inevitably slowed down the acceptance of new features somewhat and also delayed the release. However, we believe that we reached a meaningful balance between mature security practices and innovation.

On behalf of the Xen Project Hypervisor team, I would like to thank everyone for their contributions (either in the form of patches, code reviews, bug reports or packaging efforts) to the Xen Project. Please check our acknowledgement page, which recognises all those who helped make this release happen.

The source can be located in the https://xenbits.xenproject.org/gitweb/?p=xen.git;a=shortlog;h=refs/tags/RELEASE-4.10.0 tree (tag RELEASE-4.10.0) or can be downloaded as tarball from our website. For detailed download and build instructions check out the guide on building Xen 4.10

More information can be found at

What’s New in the Xen Project Hypervisor 4.9?

I am pleased to announce the release of the Xen Project Hypervisor 4.9. As always, we focused on improving code quality, security hardening as well as enabling new features. Our approach to security is also the reason why we delayed this release by 3 weeks: security issues that were discovered during the hardening phase of this release, were batched and handled using our Security Policy, which requires us to develop fixes for security issues in private and allows organisations on our pre-disclosure list to update their systems and software, before any code is made public. Consequently, we had to wait until June 20, before we could apply security fixes, build the final release candidate and test the final release candidate.

The Xen Project Hypervisor 4.9 release focuses on advanced features for embedded, automotive and native-cloud-computing use cases, enhanced boot configurations for more portability across different hardware platforms, the addition of new x86 instructions to hasten machine learning computing, and improvements to existing functionality related to the ARM® architecture, device model operation hypercall, and more.

We are also pleased to announce that Julien Grall, Senior Software Engineer at ARM, will stay release manager for Xen Project Hypervisor 4.10 release.

We grouped updates to the Xen Project Hypervisor using the following categories

  • New Features
  • Improvements to Existing Functionality
  • Multi-Release Long-Term Development

New Features

Boot Xen on EFI platforms using GRUB2 (x86): From Xen Project 4.9 and GRUB2 2.02 onwards, the Xen Project Hypervisor can be booted using the multiboot2 protocol on legacy BIOS and EFI x86 platforms. Partial support for the multiboot2 protocol was also introduced into network boot firmware (iPXE). This makes the Xen Project boot process much more flexible. Boot configurations can be changed directly from within a bootloader (without having to use text editors) and boot configurations are more portable across different platforms.

Near native latency for embedded and automotive environments: The “null” scheduler enables use-cases where every virtual CPU can be assigned to a physical CPU (commonly needed for embedded and automotive environments) removing almost all of the scheduler overheads in such environments. Usage of the “null” scheduler also guarantees significantly lower latency and more predictable performance. The new vwfi parameter for ARM (virtual Wait For Interrupt) allows fine-grained control of how the Xen Project Hypervisor handles WFI instructions. Setting vwfi to “native” reduces interrupt latency by approximately 60%. Benchmarks on Xilinx Zynq Ultrascale+ MPSoC’s have shown a maximum interrupt latency of less than 2 microseconds, which is extremely close to hardware limits, and should be small enough for the vast majority of embedded use cases.

Xen 4.9 includes new standard ABIs for sharing devices between virtual machines (including reference implementations) for a number of embedded, automotive and cloud native computing use-cases.

For embedded/automotive, a virtual sound ABI was added implementing audio playback and capture as well as volume control and the possibility to mute/unmute audio sources. In addition a new virtual display ABI for complex display devices exposing multiple framebuffers and displays has been added. Multi-touch support has been added to the virtual keyboard/mouse protocol enabling touch screens.

Xen 4.9 also introduces a Xen transport for 9pfs, which is a remote filesystem protocol originally written for Plan 9. During the Xen 4.9 release cycle, a Xen 9pfs frontend was upstreamed in the Linux kernel and a backend in QEMU. It is now possible to share a filesystem (not necessarily a block device) from a virtual machine to another, which is a requirement for adding Xen support to many container engines, such as CoreOS rkt.

The PV Calls ABI has been introduced to allow forwarding POSIX requests across guests: a POSIX function call originating from an app in a DomU can be forwarded and implemented in Dom0. For example, guest networking socket calls can be executed to Dom0, enabling a new networking model which is a natural fit for cloud-native apps.

Improvements to Existing Functionality

Xenstored optimisations: Xenstore daemons allow Dom0 and guests access to system configuration information. C-xenstored scalability limits have been increased to allow large hosts (about >1000 domains) to run efficiently. Transaction handling has been improved for better performance, smaller memory footprint and fewer transaction conflicts. Dynamic debugging capabilities have been added.

DMOP (Device Model Operation Hypercall): In Xen 4.9 the interface between Xen and QEMU was completely re-worked and consolidated. There is now only a single hypercall in Xen (the DMOP hypercall), which is carefully designed to allow the privcmd driver to audit any QEMU memory ranges and parameters that are passed to Xen via DMOP. The Linux privcmd driver enables DMOP auditing, which significantly limits the capability of a compromised QEMU to attack the hypervisor.

Alternative runtime patching and GICv3 support for ARM32: Alternative runtime patching which enables the hypervisor to apply workarounds for erratas affecting the processor and to apply optimizations specific to a CPU and GICv3 support was extended for 32-bit ARM platforms, bringing this functionality to embedded use-cases.

Intel and x86 Feature Support: The latest version of the Xen Project hypervisor adds the support of Neural Network Instructions AVX512_4VNNIW and Multiply Accumulation Single precision AVX512_4FMAPS as subfamilies of AVX512 instruction sets. With these instructions enabled in Xen for both HVM and PV guests, programs in guest OSes can take full advantage of these important instructions to speed up machine learning computing. This Xen release also further enhances VT-d Posted Interrupt (PI) optimization, Machine Check Exception(MCE) handling, and more.

System Error Detection (ARM): Xen on ARM made a step forward in reliability and serviceability with the introduction of System Error detection and reporting, a key feature for customers with highly available systems.

GCOV support: We removed the old GCOV implementation and replaced it with an updated version that supports more formats and exposes a more generic interface.

Re-work and hardening of x86 emulation code for security: Hardware-assisted virtualisation provides hypervisors with the ability to execute most privileged instructions natively and securely. However, for some boundary cases, it is still necessary to emulate x86 instructions in software. In Xen 4.9, the project completely re-worked the x86 emulation code, added support for new instructions, audited the code against security vulnerabilities and created AFL based test fuzzing tests that are regularly run against the emulator.

Updated support for Microsoft’s Hyper-V Hypervisor Top-Level Functional Specification (also known as Viridian Enlightenments): Xen implements a subset of version 5.0 of the Hyper-V Hypervisor TLFS, which enables Xen to run Windows guests at similar performance as it would run on Hyper-V. In addition, this work lays the groundwork to enable us to run Hyper-V within Xen in the future using nested virtualization.

Multi-Release Long-Term Development

This section contains large feature developments that cover several release cycles. It is intended to provide a progress update for larger features.

Transition from PVHv1 to PVHv2: Xen Project 4.8 laid the groundwork for re-architecting and simplifying PVH, focussing on the DomU guest ABI, which enabled Guest operating system developers to start porting their OSes to this mode. Support for FreeBSD is in progress, while support for Linux is committed. Xen 4.9 added Dom0 builder support and support for multiple virtual Intel I/O Advanced Programmable Interrupt Controllers (vIO APIC). PVHv2 for interrupt routing and PCI emulation is currently being peer reviewed and can be expected early in the Xen 4.10 release cycle. This lays the groundwork for a PVHv2 Dom0. For PVHv2 DomU support, PCI Passthrough and a major re-work of the xl/libxl and libvirt user interfaces for PVH have been started. Support for PVHv1 has been removed from the Xen Codebase.

Reworking the Xen-QEMU integration to protect against QEMU security vulnerabilities: In Xen Project 4.8, we embarked on an effort to re-work Xen-QEMU integration which amounts to sandboxing QEMU within Dom0. Significant progress was made in Xen 4.9 towards this goal, with the implementation of DMOP. Other changes such de-privileging QEMU in Dom0 and changes to the Linux privcmd driver have been mostly completed in Xen 4.9. Changes that are currently designed, but net yet implemented, are necessary changes to libxl and QEMU’s usage of XenStore.

Summary

Despite the shorter release cycle, the community developed several major features, and found and fixed many more bugs. Compared to Xen 4.8, which was our first fixed-term release, we have seen increased Development Velocity (in Xen 4.8 developers contributed 1245 changes – in Xen 4.9 developers contributed 1549 changes – a growth of 20%), increased Code Review activity, and more contributors (both individual and organisations contributing). For Xen 4.8 a total of 68 developers from 25 employers contributed, for Xen 4.9 a total of 86 developers from 30 employers contributed.

As in Xen 4.8, we took a security-first approach for Xen 4.9 and spent a lot of energy to improve code quality and harden security. This inevitably slowed down the acceptance of new features somewhat and also delayed the release. However, we believe that we reached a meaningful balance between mature security practices and innovation.

On behalf of the Xen Project Hypervisor team, I would like to thank everyone for their contributions (either in the form of patches, code reviews, bug reports or packaging efforts) to the Xen Project. Please check our acknowledgement page, which recognises all those who helped make this release happen.

The source can be located in the https://xenbits.xenproject.org/gitweb/?p=xen.git;a=shortlog;h=refs/tags/RELEASE-4.9.0 tree (tag RELEASE-4.9.0) or can be downloaded as tarball from our website. For detailed download and build instructions check out the guide on building Xen 4.9

More information can be found at

Xen @ Linaro Connect Europe 2013

My name is Julien Grall.  I joined the Citrix Open Source team few months ago to work on Xen on ARM with Ian Campbell and Stefano Stabellini. Since Citrix has joined the Linaro Enterprise Group (LEG), I’m also part of the virtualization team which takes care of Xen, KVM and QEMU within Linaro.

A couple of weeks ago, I have attended my first Linaro Connect Europe, held in Dublin from 8th to 12th of July.  All the major players in the ARM world came together to discuss the future of the industry and build an healthy Open Source ecosystem for ARM.

Continue reading