Category Archives: HowTo

Xen now available in CentOS 7 for ARM64 servers

A little more than a week ago at Linaro Connect SFO15 in Burlingame Jim Perrin of the CentOS project publicly announced the availability of the Xen hypervisor in CentOS 7 for ARM64 (also known as aarch64). Jim and I have been working closely with George Dunlap, maintainer of Xen in CentOS for the x86 architecture, to produce high quality Xen binaries for 64-bit ARM servers. As a result you can setup an ARM64 virtualization host with just a couple of yum commands.

CentOS 7 aarch64 is available here. Installation is trivial: download the live image, try it out, and write it to disk if you like it. You can easily extend the root partition and filesystem to match the size of your disk.

Once you have CentOS 7 up and running on your ARM64 server, you can install Xen and Libvirt with the following commands:

yum install centos-release-xen
yum update
yum install xen libvirt

If you are using AppliedMicro X-Gene, you need to add a Xen command line option to specify which serial to use. This is due to the firmware missing one piece of information. We are working with AppliedMicro to fix the issue as soon as possible. In the meantime you can edit /etc/default/grub and add the following to GRUB_CMDLINE_XEN_DEFAULT:


recreate the grub config file:

grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

Reboot and you’ll have Xen and Libvirt ready to use! Simple, right? :-)

If you want to try running a CentOS guest, just download the CentOS 7 live image, unpack it, and write a basic VM config file, using the Dom0 kernel and initramfs. For example:

disk=[ "file:/path/to/CentOS-7-aarch64-rolling.img,xvda,w" ]
name = "centos7"
vcpus = 1

Use xl to create the guest and connect to its console:

xl create -c config

Rinse and repeat as many times as you like, and you’ll have many little CentOS virtual machines keeping you company.

Xen 4.6 will be released shortly and you can count on us updating the Xen rpm in CentOS 7 shortly after. You’ll be able to install the latest and greatest Xen hypervisor release for ARM64 with a simple yum install.

At Linaro Connect I went further by showing ready to use OpenStack packages for CentOS 7 aarch64. Thanks to Anthony Perard, who produced those rpms, setting up Nova with Xen on ARM64 is just a matter of installing the packages and starting Nova services. Jim promised to have the OpenStack rpms online at in a couple of weeks. Stay tuned!

Getting Started With FreeRTOS for Xen on ARM

One of the challenges of using Xen in embedded environments is the need for core components to meet critical timing requirements. In traditional implementations engineers use real-time operating systems (RTOS) to ensure, for example, that an automobile’s brakes engage within a reasonable amount of time after the driver presses the brake pedal. It would clearly be bad if such a command were to be delayed unduly due to the car’s navigation or entertainment systems.

Over the last year, Galois has been trying to simplify one aspect of this challenge by porting an open-source RTOS, FreeRTOS, to Xen. This allows engineers to implement Xen domains that meet their own independent timing requirements. To be fully effective, this must be combined with a real-time scheduler for Xen itself, allowing for a top-to-bottom real time system. For now, we have had some success getting real-time behavior by simply pinning real-time domains to one CPU on a multi-CPU system.

In this article I’ll show you how you can get the code and how you can build your own FreeRTOS-based kernels for use on Xen/ARM systems. I’ll also provide links to technical documentation for those interested in learning more about the implementation.

As part of the community’s commitment to improving Xen for non-datacenter uses, Galois is also a member of the Xen project’s Embedded and Automotive team. Because of its key isolation and security features, flexible virtualization mode and architecture, driver disaggregation and ARM support (only 90K lines of code), Xen is a perfect fit for embedded applications.

Getting the code

The source is on GitHub and can be obtained with git:

git clone

Setting up

For the purposes of this article I’ll assume an Ubuntu Linux development system. I’ll cover how to cross-compile FreeRTOS but I’ll assume you already have Xen deployed on an ARM system. To build FreeRTOS, you’ll need:

  • An installed collection of Xen development headers corresponding to the version of Xen running on your ARM system
  • An ARM cross compiler toolchain (installable on Ubuntu under the package name arm-none-eabi-gcc)

Building FreeRTOS

The repository includes an example application in the Example/ directory. This application starts up some simple tasks and shuts down, but most of what happens in the application takes place before main() runs while the Xen services such as the console are configured.

Before we can build the application itself, we’ll need to build the FreeRTOS library. To do that, from the FreeRTOS-Xen repository, run

$ make -C Demo/CORTEX_A15_Xen_GCC

This builds FreeRTOS.a, which is everything but your application – which we’ll build and link in the next step.

NOTE: if your cross compiler name prefix isn’t arm-none-eabi-, adjust the CROSS_COMPILE variable in the Makefile accordingly. The same goes for the location of your Xen headers, which is configured by XEN_PREFIX.

Building the example application

Now we can build the application and link it against the FreeRTOS library. To do so, from the FreeRTOS-Xen repository, run

$ make -C Example/

This will build two binaries:

  • Example/Example.elf: the ELF version of your FreeRTOS application, suitable for debugging and disassembly with GDB and other tools
  • Example/Example.bin: the binary version of your FreeRTOS application, suitable for deploying as the kernel image of a Xen VM

Starting a FreeRTOS Xen guest

Once we’ve built Example.bin, we’ll need to place it somewhere on the filesystem for our ARM system and create a Xen guest configuration file, say Example.cfg, as follows (with the path to the kernel adjusted accordingly):

kernel = ".../path/to/Example.bin"
memory = 16
name = "Example"
vcpus = 1
disk = []

This port of FreeRTOS does not use multiple CPUs, so we assign only one. The application we’ve built could probably run with even less memory; the default configurable heap size is 1 MB, so even smaller memory sizes are realistic.

We can launch the VM with xl as usual:

xl create -f Example.cfg

Once the domain is up and has established a Xen console connection, we should see the application’s example tasks running and printing messages to the Xen emergency console:

Output from the example program included in the distribution of FreeRTOS for Xen on ARM

Output from the example program included in the distribution of FreeRTOS for Xen on ARM

(See the README for information about using the standard Xen console.)

Future Work & Contributions

This port is intended for research use, and we would love help with some of its missing or incomplete features. You can read a list of those in the readme in the repository and submit GitHub pull requests to contribute!

Learning More

Many more details of the port, including source code layout, Xen services, configuration parameters, memory layout, etc., can be found in the readme file in the repository. For information on FreeRTOS itself, the FreeRTOS web site provides excellent documentation on both FreeRTOS concepts and APIs.

Need help?

Contact me (Jonathan Daugherty) at jtd AT galois DOT com if you have questions and feel free to open tickets or submit pull requests on GitHub!


Using Grub 2 as a bootloader for Xen PV guests

Background: Introduction to Xen PV Bootloaders

In the very early days of Xen it was necessary for the host (domain 0)
administrator to explicitly supply a kernel (and perhaps initial
ramdisk) from the domain 0 filesystem in order to start a new guest.

This mostly worked and for some use cases, i.e. those where the host
admin wants very strict control over what each guest runs, was
desirable and remains so today.

However for other use cases it was rather inflexible since it meant
that the host administrator needed to be involved in what many
considered to be a guest administrator, or even distribution level,
decision i.e. the selection of which kernel to run, with what
parameters etc. Continue reading

Xen Project Contributor Training

A few weeks ago, I went onto a road trip to China with the aim to meet Xen Project users as well as contributors. When I was planning the trip, it became apparent that many of the developers in China are new to the project and had difficulties with Xen Project governance and how the project operated. As we also have had issues with increasing code review times due to a large influx of new developers, I put together some comprehensive training package targeted at new community members. When I started to develop the material, I realized that a lot of information related to the project’s culture, processes, conventions and communication style existed, but was not always easy to find and was not available in easily consumable way. So I spent a little more time on visualizing, adding examples, providing context, looked at communication styles and circumstances that can lead to community problems if one is not aware of them. I added some best practice that started to emerge in the project fairly recently around subjects such as design reviews, the release process, security vulnerability processes and other areas. Of course some of the material will also be applicable to other open source communities.

Coming back to the trip to China: I visited Alibaba in Hangzhou who use the Xen Project Hypervisor for many of their services including Alicloud (or Aliyun in Chinese), Citrix and Fujitsu in Nanjing each of which have large development teams there with many open source engineers including Xen Project developers and Intel in Shanghai who do much of their Virtualization, Graphics and OpenStack development on their Shanghai campus. Although the trip was hard work – after all I gave many hours of training and had many long meetings – I was impressed with the hospitality and the vibrancy of the Xen Project Developer community there. I also had a few evening dinners sampling the local cuisine, which was amazing. And I have to admit that there were a few attempts to test whether I qualify as an open source community member from a drink hardiness perspective: I think I succeeded in proving I am. A special thank you to Cai Zhimin (Fujitsu), Gui Jianfeng (Fujitsu) and Zheng Chai (Citrix) for taking some time out one evening and weekend and showing me some of the local Nanjing sights such as the Jiming Temple, the very impressive City Wall, Nanjing’s historic city center and a maple forest in its full autumn glory.

Contributor Training

You can find the actual training material below. It is divided into 3 separate modules, each of which takes approximately 2.5 hours to present. The deck is designed as reference material to read. Each training module has many examples and embedded links in it: you do need to download the PDF from slideshare or from our Developer Intro Portal to follow them though.

When I have some spare time I will create voiced over recordings of this material. Also if you have any questions, feel free to ask by contacting me via community dot manager at xenproject dot org and I will improve the material based on feedback. My plan is to keep the training material up-to-date and to modify it as new questions and new challenges arise.

Using Xen Project on OpenStack “Juno” via Libvirt

By Xing Lin

This document describes steps I took to setup a compute node based on Ubuntu 14.04 for OpenStack “juno”, using the Xen Project via libvirt approach. Openstack does not support this approach well as it is in Group C of the hypervisor support matrix for Openstack. You can hardly find any tutorial online describing this approach and this might be the first. Let’s get started!


Follow “OpenStack Installation Guide for Ubuntu 14.04″ to setup the control node and network node, following the three-node architecture with OpenStack Networking (neutron). This involves lots of configuration and could take a day or two. Check that the control node and network node is working.

Continue reading at LinuxWorld (August 5 – 7)

The community will have a booth  (#210) at the joint LinuxWorld / Next Generation Data Center event held in San Francisco, August 5- 7 at the Moscone Center. The booth will be staffed by myself as well as community members with spare time during the event.  If you are planning on attending this event and would like to help please let me know. I will be creating a schedule for community members later this year as we get closer to August.