Category Archives: HowTo

Killing Processes that Don’t Want to be Killed

This article originally appeared on

Suppose you have a program running on your system that you don’t quite trust. Maybe it’s a program submitted by a student to an automated grading system. Or maybe it’s a QEMU device model running in a Xen control domain ("domain 0" or “dom0”), and you want to make sure that even if an attacker from a rogue virtual machine manages to take over the QEMU process, they can’t do any further harm. There are many things you want to do as far as restricting its ability to do mischief. But one thing in particular you probably want to do is to be able to reliably kill the process once you think it should be done. This turns out to be quite a bit more tricky than you’d think.

Continue reading

Xen on ARM interrupt latency

Xen on ARM is becoming more and more widespread in embedded environments. In these contexts, Xen is employed as a single solution to partition the system into multiple domains, fully isolated from each other, and with different levels of trust.

Every embedded scenario is different, but many require real-time guarantees. It comes down to interrupt latency: the hypervisor has to be able to deliver interrupts to virtual machines within a very small timeframe. The maximum tolerable lag changes on a case by case basis, but it should be in the realm of nanoseconds and microseconds, not milliseconds.

Xen on ARM meets these requirements in a few different ways. Firstly, Xen comes with a flexible scheduler architecture. Xen includes a set of virtual machine schedulers, including RTDS, a soft real-time scheduler, and ARINC653, a hard real-time scheduler. Users can pick the ones that perform best for their use-cases. However, if they really care about latency, the best option is to have no schedulers at all and use a static assignment for virtual cpus to physical cpus instead. There are no automatic ways to do that today, but it is quite easy to achieve with the vcpu-pin command:

Usage: xl vcpu-pin [domain-id] [vcpu-id] [pcpu-id]

For example, in a system with four physical cpus and two domains with two vcpus each, a user can get a static configuration with the following commands:

xl vcpu-pin 0 0 0
xl vcpu-pin 0 1 1
xl vcpu-pin 1 0 2
xl vcpu-pin 1 1 3

As a result, all vcpus are pinned to different physical cpus. In such a static configuration, the latency overhead introduced by Xen is minimal. Xen always configures interrupts to target the cpu that is running the virtual cpu that should receive the interrupt. Thus, the overhead is down to just the time that it takes to execute the code in Xen to handle the physical interrupt and inject the corresponding virtual interrupt to the vcpu.

For my measurements, I used a Xilinx Zynq Ultrascale+ MPSoC, an excellent board with four Cortex A53 cores and a GICv2 interrupt controller. I installed Xen 4.9 unstable (changeset 55a04feaa1f8ab6ef7d723fbb1d39c6b96ad184a) and Linux 4.6.0 as Dom0. I ran tbm as a guest, which is a tiny baremetal application that programs timer events in the future, then, after receiving them, checks the current time again to measure the latency. tbm uses the virtual timer for measurements, however, the virtual timer interrupt is handled differently compared to any other interrupts in Xen. Thus, to make the results more generally applicable, I modified tbm to use the physical timer interrupt instead. I also modified Xen to forward physical timer interrupts to guests.

Keeping in mind that the native interrupt latency is about 300ns on this board, these are the results on Xen in nanoseconds:

4850 4810 7030 4980

AVG is the average latency, MIN is the minimum, MAX is the maximum and WARM_MAX is the maximum latency observed after discarding the first few interrupts to warm the caches. For real-time considerations, the number to keep in mind is WARM_MAX, which is 5000ns (when using static vcpu assignments).

This excellent result is small enough for most use cases, including piloting a flying drone. However, it can be further improved by using the new vwfi Xen command line option. Specifically, when vcpus are statically assigned to physical cpus using vcpu-pin, it makes sense to pass vwfi=native to Xen: it tells the hypervisor not to trap wfi and wfe commands, which are ARM instructions for sleeping. If no other vcpus can ever be scheduled on a given physical cpu, then we might as well let the guest put the cpu to sleep. Passing vwfi=native, the results are:

1850 1680 2650 1950

With this configuration, the latency is only 2 microseconds, which is extremely close to the hardware limits, and should be small enough for the vast majority of use cases. vwfi was introduced recently, but it has been backported to all the Xen stable trees.

In addition to vcpu pinning and vwfi, the third key parameter to reduce interrupt latency is unexpectedly simple: the DEBUG kconfig option in Xen. DEBUG is enabled by default in all cases except for releases. It adds many useful debug messages and checks, at the cost of increased latency. Make sure to disable it in production and when doing measurements.

Xen now available in CentOS 7 for ARM64 servers

A little more than a week ago at Linaro Connect SFO15 in Burlingame Jim Perrin of the CentOS project publicly announced the availability of the Xen hypervisor in CentOS 7 for ARM64 (also known as aarch64). Jim and I have been working closely with George Dunlap, maintainer of Xen in CentOS for the x86 architecture, to produce high quality Xen binaries for 64-bit ARM servers. As a result you can setup an ARM64 virtualization host with just a couple of yum commands.

CentOS 7 aarch64 is available here. Installation is trivial: download the live image, try it out, and write it to disk if you like it. You can easily extend the root partition and filesystem to match the size of your disk.

Once you have CentOS 7 up and running on your ARM64 server, you can install Xen and Libvirt with the following commands:

yum install centos-release-xen
yum update
yum install xen libvirt

If you are using AppliedMicro X-Gene, you need to add a Xen command line option to specify which serial to use. This is due to the firmware missing one piece of information. We are working with AppliedMicro to fix the issue as soon as possible. In the meantime you can edit /etc/default/grub and add the following to GRUB_CMDLINE_XEN_DEFAULT:


recreate the grub config file:

grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

Reboot and you’ll have Xen and Libvirt ready to use! Simple, right? 🙂

If you want to try running a CentOS guest, just download the CentOS 7 live image, unpack it, and write a basic VM config file, using the Dom0 kernel and initramfs. For example:

disk=[ "file:/path/to/CentOS-7-aarch64-rolling.img,xvda,w" ]
name = "centos7"
vcpus = 1

Use xl to create the guest and connect to its console:

xl create -c config

Rinse and repeat as many times as you like, and you’ll have many little CentOS virtual machines keeping you company.

Xen 4.6 will be released shortly and you can count on us updating the Xen rpm in CentOS 7 shortly after. You’ll be able to install the latest and greatest Xen hypervisor release for ARM64 with a simple yum install.

At Linaro Connect I went further by showing ready to use OpenStack packages for CentOS 7 aarch64. Thanks to Anthony Perard, who produced those rpms, setting up Nova with Xen on ARM64 is just a matter of installing the packages and starting Nova services. Jim promised to have the OpenStack rpms online at in a couple of weeks. Stay tuned!

Getting Started With FreeRTOS for Xen on ARM

One of the challenges of using Xen in embedded environments is the need for core components to meet critical timing requirements. In traditional implementations engineers use real-time operating systems (RTOS) to ensure, for example, that an automobile’s brakes engage within a reasonable amount of time after the driver presses the brake pedal. It would clearly be bad if such a command were to be delayed unduly due to the car’s navigation or entertainment systems.

Over the last year, Galois has been trying to simplify one aspect of this challenge by porting an open-source RTOS, FreeRTOS, to Xen. This allows engineers to implement Xen domains that meet their own independent timing requirements. To be fully effective, this must be combined with a real-time scheduler for Xen itself, allowing for a top-to-bottom real time system. For now, we have had some success getting real-time behavior by simply pinning real-time domains to one CPU on a multi-CPU system.

In this article I’ll show you how you can get the code and how you can build your own FreeRTOS-based kernels for use on Xen/ARM systems. I’ll also provide links to technical documentation for those interested in learning more about the implementation.

As part of the community’s commitment to improving Xen for non-datacenter uses, Galois is also a member of the Xen project’s Embedded and Automotive team. Because of its key isolation and security features, flexible virtualization mode and architecture, driver disaggregation and ARM support (only 90K lines of code), Xen is a perfect fit for embedded applications.

Getting the code

The source is on GitHub and can be obtained with git:

git clone

Setting up

For the purposes of this article I’ll assume an Ubuntu Linux development system. I’ll cover how to cross-compile FreeRTOS but I’ll assume you already have Xen deployed on an ARM system. To build FreeRTOS, you’ll need:

  • An installed collection of Xen development headers corresponding to the version of Xen running on your ARM system
  • An ARM cross compiler toolchain (installable on Ubuntu under the package name arm-none-eabi-gcc)

Building FreeRTOS

The repository includes an example application in the Example/ directory. This application starts up some simple tasks and shuts down, but most of what happens in the application takes place before main() runs while the Xen services such as the console are configured.

Before we can build the application itself, we’ll need to build the FreeRTOS library. To do that, from the FreeRTOS-Xen repository, run

$ make -C Demo/CORTEX_A15_Xen_GCC

This builds FreeRTOS.a, which is everything but your application – which we’ll build and link in the next step.

NOTE: if your cross compiler name prefix isn’t arm-none-eabi-, adjust the CROSS_COMPILE variable in the Makefile accordingly. The same goes for the location of your Xen headers, which is configured by XEN_PREFIX.

Building the example application

Now we can build the application and link it against the FreeRTOS library. To do so, from the FreeRTOS-Xen repository, run

$ make -C Example/

This will build two binaries:

  • Example/Example.elf: the ELF version of your FreeRTOS application, suitable for debugging and disassembly with GDB and other tools
  • Example/Example.bin: the binary version of your FreeRTOS application, suitable for deploying as the kernel image of a Xen VM

Starting a FreeRTOS Xen guest

Once we’ve built Example.bin, we’ll need to place it somewhere on the filesystem for our ARM system and create a Xen guest configuration file, say Example.cfg, as follows (with the path to the kernel adjusted accordingly):

kernel = ".../path/to/Example.bin"
memory = 16
name = "Example"
vcpus = 1
disk = []

This port of FreeRTOS does not use multiple CPUs, so we assign only one. The application we’ve built could probably run with even less memory; the default configurable heap size is 1 MB, so even smaller memory sizes are realistic.

We can launch the VM with xl as usual:

xl create -f Example.cfg

Once the domain is up and has established a Xen console connection, we should see the application’s example tasks running and printing messages to the Xen emergency console:

Output from the example program included in the distribution of FreeRTOS for Xen on ARM

Output from the example program included in the distribution of FreeRTOS for Xen on ARM

(See the README for information about using the standard Xen console.)

Future Work & Contributions

This port is intended for research use, and we would love help with some of its missing or incomplete features. You can read a list of those in the readme in the repository and submit GitHub pull requests to contribute!

Learning More

Many more details of the port, including source code layout, Xen services, configuration parameters, memory layout, etc., can be found in the readme file in the repository. For information on FreeRTOS itself, the FreeRTOS web site provides excellent documentation on both FreeRTOS concepts and APIs.

Need help?

Contact me (Jonathan Daugherty) at jtd AT galois DOT com if you have questions and feel free to open tickets or submit pull requests on GitHub!


Using Grub 2 as a bootloader for Xen PV guests

Background: Introduction to Xen PV Bootloaders

In the very early days of Xen it was necessary for the host (domain 0)
administrator to explicitly supply a kernel (and perhaps initial
ramdisk) from the domain 0 filesystem in order to start a new guest.

This mostly worked and for some use cases, i.e. those where the host
admin wants very strict control over what each guest runs, was
desirable and remains so today.

However for other use cases it was rather inflexible since it meant
that the host administrator needed to be involved in what many
considered to be a guest administrator, or even distribution level,
decision i.e. the selection of which kernel to run, with what
parameters etc. Continue reading

Xen Project Contributor Training

A few weeks ago, I went onto a road trip to China with the aim to meet Xen Project users as well as contributors. When I was planning the trip, it became apparent that many of the developers in China are new to the project and had difficulties with Xen Project governance and how the project operated. As we also have had issues with increasing code review times due to a large influx of new developers, I put together some comprehensive training package targeted at new community members. When I started to develop the material, I realized that a lot of information related to the project’s culture, processes, conventions and communication style existed, but was not always easy to find and was not available in easily consumable way. So I spent a little more time on visualizing, adding examples, providing context, looked at communication styles and circumstances that can lead to community problems if one is not aware of them. I added some best practice that started to emerge in the project fairly recently around subjects such as design reviews, the release process, security vulnerability processes and other areas. Of course some of the material will also be applicable to other open source communities.

Coming back to the trip to China: I visited Alibaba in Hangzhou who use the Xen Project Hypervisor for many of their services including Alicloud (or Aliyun in Chinese), Citrix and Fujitsu in Nanjing each of which have large development teams there with many open source engineers including Xen Project developers and Intel in Shanghai who do much of their Virtualization, Graphics and OpenStack development on their Shanghai campus. Although the trip was hard work – after all I gave many hours of training and had many long meetings – I was impressed with the hospitality and the vibrancy of the Xen Project Developer community there. I also had a few evening dinners sampling the local cuisine, which was amazing. And I have to admit that there were a few attempts to test whether I qualify as an open source community member from a drink hardiness perspective: I think I succeeded in proving I am. A special thank you to Cai Zhimin (Fujitsu), Gui Jianfeng (Fujitsu) and Zheng Chai (Citrix) for taking some time out one evening and weekend and showing me some of the local Nanjing sights such as the Jiming Temple, the very impressive City Wall, Nanjing’s historic city center and a maple forest in its full autumn glory.

Contributor Training

You can find the actual training material below. It is divided into 3 separate modules, each of which takes approximately 2.5 hours to present. The deck is designed as reference material to read. Each training module has many examples and embedded links in it: you do need to download the PDF from slideshare or from our Developer Intro Portal to follow them though.

When I have some spare time I will create voiced over recordings of this material. Also if you have any questions, feel free to ask by contacting me via community dot manager at xenproject dot org and I will improve the material based on feedback. My plan is to keep the training material up-to-date and to modify it as new questions and new challenges arise.