Monthly Archives: January 2015

Xen Project @ FOSDEM

Going to FOSDEM’15? Well, you want to check out the schedule of the Virtualization & IaaS devroom then, and make sure you do not miss the talks about Xen.

Here they are the talks, in some more details:

Last but certainly not least, there will be a Xen-Project booth, where you can meet members of the Xen community as well as enjoying some demos.

The booth will be in building K, on level 1.

We also have a handy guide available, that you may want to print out before going to the event.

CES 2015: Smart Cars are the New Smart Phone

This is a reprint of the following article by Alex Agizim, VP, CTO Embedded Systems at GlobalLogic

“Smart car” technology had a huge presence at CES 2015, from BMW’s 360-degree collision avoidance and parking assist features to Audi’s Human Machine Interface (HMI) that connects to an iPhone or Android device. And with both Apple and Google jumping into the market with their CarPlay and Android Auto IVI systems, the automotive industry is on the brink of some significant changes.

For example, thanks to new developments in open source virtualization, OEMs and car manufacturers are closer than ever to achieving a secure, flexible, robust, and customizable integrated cockpit — one that keeps drivers safe while meeting consumers’ connected car expectations. Already well-known for providing security, stability, and isolation in the datacenter, automotive virtualization is gaining wider attention due to additional hardening and new support for ARM.

While this is certainly exciting, virtualization remains a roadblock to some in the smart car industry. I personally had the opportunity to demonstrate GlobalLogic’s Nautilus platform for automotive virtualization at GENIVI’s CES demo and networking event. Leveraging a TI J6 SoC, I demo’d a dual-screen virtual cockpit with one screen emulating a Linux-powered driver information display, and the other screen emulating an Android-powered IVI system. The entire configuration ran on Xen Project Hypervisor 4.5 with three domains: Dom0 (thin control), DomU (Linux), and DomU (Android).

During the demo, I showcased how Nautilus achieves an overall system boot time of 8 seconds, an early RVC of 1.5 seconds, and secure and reliable peripheral sharing (including GPUs). Most importantly, I demonstrated how even if the Android virtual machine crashes, it has absolutely no influence on the mission-critical Linux virtual machine. With Nautilus automotive software, developers can host a number of VMs that are completely sandboxed from each other, thereby ensuring that all vehicle services will continue to operate even if one specific component fails.

The demo was well-received by GENIVI’s attendees, and I got the impression that many Tier 1 OEMs were thinking about using virtualization in their next-gen platforms. This is a huge milestone because, up until very recently, virtualization had a bad rep in the automotive industry. Previous attempts at virtualization using ARM A9 architecture ultimately failed because there was no hardware support for it. Many were also highly reluctant to use open source technology because it lacked proper compliance to strict auto industry regulations. But with platforms like Nautilus, developers can leverage cutting-edge open source technology that is ISO 26262 certification ready to create secure and reliable automotive virtualization experiences.

In fact, GlobalLogic’s goal is to make Nautilus part of the reference Automotive Grade Linux (AGL) software, an open source project that is developing a common, Linux-based software stack for the connected car. We are also a founding leader for Xen Project’s Embedded and Automotive initiative. GlobalLogic is working to add the Xen-based technology to the AGL spec and is further developing the platform’s real-time scheduling and peripheral sharing features to improve the use of a single physical CPU for multiple guest OSes and peripheral devices. We’ll soon be extending QNX and Tizen IVI 3.0 support to improve the functionality of other features. Finally, we are also expanding Nautilus to support even more SoCs in the next six months, such as Renesas R-Car H2/M2, which offers hardware support for virtualization.

Based on my work with the Nautilus platform and my observations of the general automotive industry, I wouldn’t be surprised to see the first PoCs for automotive virtualization coming out of China and Japan later this year. The momentum behind smart car technology development is very strong right now, and I’m excited to see what happens when automotive OEMs finally start taking advantage of virtualization’s many possibilities.

Future of Xen Project: Video Spotlight Interview with Intel’s Donald Dugger

Intel’s Virtualization Architect Donald Dugger started working on Xen Project software eight years ago. We recently interviewed Don to find out why Intel continues to support, contribute and invest in the Xen Project. One of the first companies to contribute to hardware-assisted virtualization, today Intel remains equally focused on actively promoting open source virtualization. The company continually adds new virtualization features in its CPUs and is constantly evolving its virtualization support. Improved cache monitoring technology, which provides faster processing and better utilization to resolve the “noisy neighbor” dilemma when hosting large, resource-hungry data sets, is the latest contribution from the world’s largest chip company. Don spoke to eWeek about this new feature last week for the release of Xen Project Hypervisor version 4.5.

In this video, Don discusses the pressure data centers face today to reduce costs and achieve more efficient use of hardware. Open source Xen provides a very secure, efficient and cost-effective way to solve these problems and allows organizations to do more with less. Don also talks about the key role open source virtualization plays in cloud computing, which is poised for continued growth as datacenters struggle with capacity and resource availability. Don says Intel remains deeply committed to the Project to best service customers running a cloud environment based on Xen virtualization and utilizing Intel hardware.

Less is More in the New Xen Project 4.5 Release

If we used code-names, the Xen 4.5 release should be called Panda on Diet! We have 78K new code with 141K deleted. In effect this release has -63KLOC code than the previous one.

The net effect of a skinnier Xen Project Hypervisor code base is increased usability, simplicity and innovation. This is all by design and one of many steps we’ll continue to take to fine-tune our development and release cycle.

For example, we shed the Python toolstack – including xend which we deprecated in 4.3. This comprised the majority of the code deleted in today’s release, which is a big boon for developers who now have less code to maintain and can spend more time on new features.

And 4.5 is more feature-rich than any release in Xen Project’s history.


Today we are announcing specific patches in Xen Project Hypervisor 4.5 that span from architecture (x86 and ARM), platforms (different ARM, AMD or Intel boards), to generic code. The release also creates new opportunity to incorporate Xen virtualization into software stacks in markets like embedded computing, automotive, drones, avionics and more.

Virtualization and open source are more relevant than ever in today’s evolving, more software-centric data center too. New developments with hyper scale-out computing, Internet of Things, NFV/SDN, and next-generation ARM-based products are driving increased demand for better resource sharing and utilization with enough flexibility to efficiently grow well into the future. What isn’t likely to change anytime soon is the diversity of OSes, multi-tenant architectures, security concerns and storage and network challenges that cloud providers and enterprises must contend with to run their applications. Undeniably, abstraction at the VM level is necessary for superior performance and security in these environments.

Despite these impressive and rapid changes, or perhaps because of them, Xen Project developers are motivated to continually stay ahead of the market with performance, speed, agility and security improvements. Our traditional customers also inspire us; organizations such as Alibaba, Amazon Web Services, IBM Softlayer, Rackspace, Oracle and others are some of the most savvy and innovative users around.

To learn more about the release and for ease of reading, I’ve grouped the summary of updates into four major categories:

  • Hypervisor specific

  • Toolstack

  • External users of toolstack

  • Linux, FreeBSD, and other OSes that can utilize the new features.

x86 Hypervisor-Specific Updates

On the x86 side, development has focused on improving performance on various fronts:

  • The HPET has been modified to provide faster and better resolution values.

  • Memory is scrubbed in parallel on bootup, giving a huge time boost for large-scale machines (1TB or more).

  • PVH initial domain support for Intel has been added and now supports running as dom0 and FreeBSD with Linux platforms. PVH is an extension to the classic Xen Project Paravirtualization (PV) that uses the hardware virtualization extensions available on modern x86 processor servers. Requiring no additional support other than the hypervisor, PVH boots as the first guest and takes on the responsibilities of the initial domain known as dom0. This means Xen Project Hypervisor is able to take advantage of contemporary hardware features like virtual machine extensions (VMX) to significantly expedite execution of the initial domain. Instead of asking the hypervisor to handle certain operations, the dom0 can execute operations natively without compromising security. For more background, Virtualization Spectrum is an excellent introduction to PVH.

  • Lower interrupt latency for PCI passthrough on large-scale machines (more than 2 sockets).

  • Multiple IO-REQ services for guests, which is a technique to have many QEMUs assigned for one domain. This allows speed up of guests operation by having multiple backends (QEMUs) deal with different emulations.

We also expanded support for:

  • Soft affinity for vCPUs: Xen has had NUMA- aware scheduling ( since 4.3. In Xen 4.5, we build on that to make it more general and useful on non-NUMA systems. In fact, it is now possible for the sysadmin to define an arbitrary set of physical CPUs on which vCPUs prefer to run on, and Xen will try as hard as possible to follow this indication.

  • Security improvements – guest introspection expansion: VM introspection using Intel EPT / AMD RVI hardware virtualization functionality builds on Xen Project Hypervisor Memory Inspection APIs introduced in 2011. This addresses a number of security issues from outside the guest OS, without relying on functionality that can be rendered unreliable by advanced malware. The approach works by auditing access of sensitive memory areas using HW support in guests with minimal overhead and allows control software running within a dedicated VM to allow or deny attempts to access sensitive memory based on policy and security heuristics. You can find an excellent introduction on the topic of VM introspection here and a video on Youtube (a recording of this presentation) explaining the new functionality in Xen 4.5.

  • Serial support for debug purposes. This covers PCIe cards (Oxford ones) and newer Broadcom ones found on blades.

  • Experimental support for Real-Time Scheduling: a new, multicore-enabled, real-time scheduler, called RTDS is part of Xen 4.5 as an experimental feature. Virtualization will soon become the norm rather than the exception in automotive, avionics, mobile and multimedia, and other fields where predictability and high-end, real-time support are critical. Xen wants to play a big role in this, and this new scheduler will allow for such, which is why we introduced it in 4.5 while still under development. More information here: Youtube video, Linux Foundation presentation and related blog.

Intel Hypervisor-Specific Updates

  • Broadwell Supervisor Mode Access Prevention. This LWN article has an excellent explanation of it – but a short summary is that it restricts the kernel from accessing the user-space pages. This feature in Xen also added alternative assembler support to patch the hypervisor during run-time (so that we won’t be running these operations on older hardware).

  • Haswell Server Cache QoS Monitoring, aka Intel Resource Director Technology, is a “new area of architecture extension that seeks to provide better information and control of applications running on Intel processors. The feature, ” … documented in the Software Developers’ Manual, relates to monitoring application thread LLC usage, to provide a means of directing such usage and provide more information on the amount of memory traffic out of the LLC,” according to xen-devel.

  • SandyBridge (vAPIC) extensions.  Xen 4.3 added support for VT-d Posted Interrupts, and  in Xen 4.5 we added extensions for PVHVM guests to take advantage of VT-d Posted Interrupts. Instead of using vector callback, the guest can utilize the vAPIC to lower its VMEXIT overhead, leading to lower interrupt latency and performance improvements for I/O intensive workloads in PVHMM guests.

AMD Hypervisor-Specific Updates

  • Fixes in the microcode loading.

  • Data Breakpoint Extensions and further MSR masking support for Kabini, Kaveri and newer. This allows “.. to specify cpuid masks to help with cpuid levelling across a pool of hosts,” from the xen-command-line manual.

ARM Hypervisor-Specific Updates

The ARM ecosystem operates differently than the x86 architecture – in which ARM licensees design new chipsets and features and OEMs manufacture platforms based on these specifications. OEMs designing ARM-based platforms determine what they need on the SoC – that is the System On Chip. As such, they can selectively enable or disable certain functionality that they consider important (or unimportant). ARM provides the Intellectual Property (IP) and standards from which OEMs can further specialize and optimize. Therefore the features Xen Project Hypervisor supports on ARM are not for a specific platform – but rather for functionality SoCs provide. New updates include:

  • Support for up to 1TB for guests.

  • The Generic Interrupt Controller (GIC) v3 is supported in Xen 4.5. v3 is very important because it introduces support for Message Signaled Interrupts (MSI), emulation of GICv3 for guests – and most importantly – for more than 8 CPUS. Many of the new features are not used by Xen yet but the driver is on par with v2.

  • Power State Coordination Interface 0.2 (PSCI) is important in embedded environments where power consumption needs to be kept to the absolute minimum. It allows us to power down/up CPUS, suspend them, etc.

  • UEFI booting. On ARM64 servers both U-Boot and UEFI can be used to boot the OS.

  • IOMMU support (SMMUv1). For isolation between guests, ARM platforms can come with an IOMMU chipset based on the SMMU specification.

  • Super Pages (2MB) support in Xen. Using super pages for the guest pseudo-physical to physical translation tables significantly improves overall guest performance.

  • Passthrough – the PCI passthrough features did not make it on time, but doing passthrough of MMIO regions did. In the ARM world, it is quite common to have no PCIe devices and to only access devices using MMIO regions. As such this feature allows us to have driver domains be in charge of network or storage devices.

  • Interrupt latency reduction: By removing maintenance interrupts, we get rid of an expensive trap into Xen for each interrupt EOI. Please see Stefano’s slides.

With these new features, the following motherboards are now supported in Xen Project Hypervisor 4.5:

  • AMD Seattle

  • Broadcom 7445D0 A15

  • Midway (Calxeda)

  • Vexpress (ARM Ltd.)

  • OMAP5, DRA7 (Texas Instrument)

  • Exynos5250 (Exynos 5 Dual), Odroid-Xu, and Exynos 5420 (Exynos Octa) (Samsung SoC for Arndale and various smartphones and tablets)

  • SunXI (AllWinner), aka A20/A21, CubieTruck, CubieBoard

  • Mustang (Applied Micro-X-Gene, the ARMv8 SoC)

  • McDivitt aka HP Moonshot cartridge (Applied Micro X-Gene)

  • The Xen Project also maintains this list of ARM boards that work with Xen Project software.

Toolstack Updates

Xen Project software is now using a C-based toolstack called xl or libxl, replacing the obsolete Python toolstack called xend.  This more modern architecture is easier to easier maintain, and users will not be affected by the move since xm and xl offer feature parity. In fact, the switch greatly simplifies managing Xen instances as other toolstack, such as libvirt are C based and less complex. libvirt and XAPI are now using libxl as well. For more background, check out our new hands-on tutorial “XM to XL: A Short, but Necessary, Journey.”

Additional toolstack changes include:

  • VM Generation ID. This allows Windows 2012 Server and later active directory domain controllers to be migrated.

  • Remus initial support provides high availability by check pointing guests states at high frequency.

  • Libxenlight (libxl) JSON infrastructure support. This allows libxenlight to use JSON to communicate with other toolstacks.

  • Libxenlight to keep track of domain configuration. It now uses the JSON infrastructure to keep track of domain configuration. The is feature parity with Xend.

  • Systemd support. This allows one source base to contain the systemd files, which can be used by various distributions instead of them having to generate them.

  • vNUMA,while still in progress,  is coming along nicely thanks to sponsorship from . Virtual NUMA allows Xen to expose to the guest the NUMA topology (either based on the host or made-up) for the guest.

On the libvirt side, changes include:

  • PCI/SR-IOV passthrough, including hot{un}plug

  • Migration support

  • Improved concurrency through job support in the libxl driver – no more locking entire driver when modifying a domain

  • Improved domxml-{to,from}-native support, e.g. for converting between xl config and libvirt domXML and vise-versa

  • PV console support

  • Improved qdisk support

  • Support for <interface type=’network’> – allows using libvirt-managed networks in the libxl driver

  • Support PARAVIRT and ACPI shutdown flags

  • Support PARAVIRT reboot flag

  • Support for domain lifecycle event configuration, e.g. on_crash, on_reboot, etc

  • A few improvements for ARM

  • Lots of bug fixes

QEMU Updates

Xen Project 4.5 will ship with QEMU v2.0 and SeaBIOS v1.7.5 with the following updates:

  • Bigger PCI hole in QEMU via the mmio_hole parameter in guest config. This allows users to pack more legacy PCI devices for passthrough in an guest.

  • QEMU is now built for ARM providing backend support for framebuffer (VNC).


The 4.5 release also takes advantage of new features in Linux and FreeBSD such as PVH support (which is considered experimental)


With 43 major new features, 4.5 includes the most updates in our project’s history. That’s not even counting 22 new enablers in up-streams (Linux and QEMU). The Project is also taking a more holistic, proactive approach to managing dependencies such as Linux and QEMU, as well as downstream functionality such as libvirt. In 2015, we plan to build on this even further up the stack to include OpenStack and other key projects. For the first time, our Project’s development process is robust, active and mature enough to systematically focus on these strategic growth opportunities. It also reflects enhanced responsiveness to community feedback; for example, we’re improving usability and performing broader testing for specific use cases with new releases.

During this development and release we’ve seen a steady influx of folks helping, contributing, testing and reporting. As the Release Manager, I would like to thank everybody and call out major contributions coming from AMD, Bitdefender, Citrix, Fujitsu, GlobalLogic, Intel, Linaro, Oracle, SuSE and Cavium, as well as several individual and academic institutions.

The sources are located in the git tree or one can download the tarball:

  • xen: with a recent enough git (>= just pull from the proper tag (RELEASE-4.5.0) from the main repo directly:

  • git clone -b RELEASE-4.5.0 git://

  • With an older git version (and/or if that does not work, e.g., complaining with a message like this: Remote branch RELEASE-4.5.0 not found in upstream origin, using HEAD instead), do the following:

  • git clone git:// ; cd xen ; git checkout RELEASE-4.5.0

  • tarball: here it is a 4.5.0 and its signature.

Release Documentation can be found on our wiki.

Using Grub 2 as a bootloader for Xen PV guests

Background: Introduction to Xen PV Bootloaders

In the very early days of Xen it was necessary for the host (domain 0)
administrator to explicitly supply a kernel (and perhaps initial
ramdisk) from the domain 0 filesystem in order to start a new guest.

This mostly worked and for some use cases, i.e. those where the host
admin wants very strict control over what each guest runs, was
desirable and remains so today.

However for other use cases it was rather inflexible since it meant
that the host administrator needed to be involved in what many
considered to be a guest administrator, or even distribution level,
decision i.e. the selection of which kernel to run, with what
parameters etc. Continue reading