Tag Archives: FreeRTOS

Getting Started With FreeRTOS for Xen on ARM

One of the challenges of using Xen in embedded environments is the need for core components to meet critical timing requirements. In traditional implementations engineers use real-time operating systems (RTOS) to ensure, for example, that an automobile’s brakes engage within a reasonable amount of time after the driver presses the brake pedal. It would clearly be bad if such a command were to be delayed unduly due to the car’s navigation or entertainment systems.

Over the last year, Galois has been trying to simplify one aspect of this challenge by porting an open-source RTOS, FreeRTOS, to Xen. This allows engineers to implement Xen domains that meet their own independent timing requirements. To be fully effective, this must be combined with a real-time scheduler for Xen itself, allowing for a top-to-bottom real time system. For now, we have had some success getting real-time behavior by simply pinning real-time domains to one CPU on a multi-CPU system.

In this article I’ll show you how you can get the code and how you can build your own FreeRTOS-based kernels for use on Xen/ARM systems. I’ll also provide links to technical documentation for those interested in learning more about the implementation.

As part of the community’s commitment to improving Xen for non-datacenter uses, Galois is also a member of the Xen project’s Embedded and Automotive team. Because of its key isolation and security features, flexible virtualization mode and architecture, driver disaggregation and ARM support (only 90K lines of code), Xen is a perfect fit for embedded applications.

Getting the code

The source is on GitHub and can be obtained with git:

git clone https://github.com/GaloisInc/FreeRTOS-Xen.git

Setting up

For the purposes of this article I’ll assume an Ubuntu Linux development system. I’ll cover how to cross-compile FreeRTOS but I’ll assume you already have Xen deployed on an ARM system. To build FreeRTOS, you’ll need:

  • An installed collection of Xen development headers corresponding to the version of Xen running on your ARM system
  • An ARM cross compiler toolchain (installable on Ubuntu under the package name arm-none-eabi-gcc)

Building FreeRTOS

The repository includes an example application in the Example/ directory. This application starts up some simple tasks and shuts down, but most of what happens in the application takes place before main() runs while the Xen services such as the console are configured.

Before we can build the application itself, we’ll need to build the FreeRTOS library. To do that, from the FreeRTOS-Xen repository, run

$ make -C Demo/CORTEX_A15_Xen_GCC

This builds FreeRTOS.a, which is everything but your application – which we’ll build and link in the next step.

NOTE: if your cross compiler name prefix isn’t arm-none-eabi-, adjust the CROSS_COMPILE variable in the Makefile accordingly. The same goes for the location of your Xen headers, which is configured by XEN_PREFIX.

Building the example application

Now we can build the application and link it against the FreeRTOS library. To do so, from the FreeRTOS-Xen repository, run

$ make -C Example/

This will build two binaries:

  • Example/Example.elf: the ELF version of your FreeRTOS application, suitable for debugging and disassembly with GDB and other tools
  • Example/Example.bin: the binary version of your FreeRTOS application, suitable for deploying as the kernel image of a Xen VM

Starting a FreeRTOS Xen guest

Once we’ve built Example.bin, we’ll need to place it somewhere on the filesystem for our ARM system and create a Xen guest configuration file, say Example.cfg, as follows (with the path to the kernel adjusted accordingly):

kernel = ".../path/to/Example.bin"
memory = 16
name = "Example"
vcpus = 1
disk = []

This port of FreeRTOS does not use multiple CPUs, so we assign only one. The application we’ve built could probably run with even less memory; the default configurable heap size is 1 MB, so even smaller memory sizes are realistic.

We can launch the VM with xl as usual:

xl create -f Example.cfg

Once the domain is up and has established a Xen console connection, we should see the application’s example tasks running and printing messages to the Xen emergency console:

Output from the example program included in the distribution of FreeRTOS for Xen on ARM

Output from the example program included in the distribution of FreeRTOS for Xen on ARM

(See the README for information about using the standard Xen console.)

Future Work & Contributions

This port is intended for research use, and we would love help with some of its missing or incomplete features. You can read a list of those in the readme in the repository and submit GitHub pull requests to contribute!

Learning More

Many more details of the port, including source code layout, Xen services, configuration parameters, memory layout, etc., can be found in the readme file in the repository. For information on FreeRTOS itself, the FreeRTOS web site provides excellent documentation on both FreeRTOS concepts and APIs.

Need help?

Contact me (Jonathan Daugherty) at jtd AT galois DOT com if you have questions and feel free to open tickets or submit pull requests on GitHub!


Python Migration Utility for xend Virtual Machines to XCP

A new python based migration utility for migrating your xend virtual machines to XCP  is now available at:


It can generate XVAs from your xend based HVM and paravirtualised virtual machines.
It can also steam your VM directly to a XenServer/XCP host over HTTP or HTTPS.

Please read the README for pointers and guidelines on how to prepare your VMs for migration to XCP. Most PV based virtual machines will need some preparation work.

XEN-HA Cluster Demonstration on YouTube

Andreas Ender from SECURE_iT sent me this excellent demonstration video…

Georg Dörn from www.its-doern.at has built a XEN cluster solution.

This Video shows you an RDP session while live migration of a Windows 2008 R2 terminal server running as a virtual machine on a XEN-HA cluster.

Within the RDP session a video is playing while live migrating the terminal server vm from one XEN-HA cluster node to the other node and back again without any downtime.

Only one ping is lost while migrating. You will see a very short stop playing the video at 1:06 and 2:12.

The XEN-HA Cluster is based on two identical Debian 5.0 XEN Hosts with drbd (raid 1 with sata disks and sync with 2 x gbit nic) and heartbeat.
The whole system runs with open source software! This XEN-HA cluster is not any experimental system. It works as a production system at many customers from us.

Project Remus 0.9 Released

From xen-devel:

It’s my pleasure to announce the release of Remus 0.9!

Remus provides comprehensive fault tolerance for Xen virtual machines. If the physical machine hosting your VM fails, the backup can take over instantly, as if you had migrated it to the backup at the instant before the failure occured. There’s no need for recovery, because the backup is always completely up to date. Furthermore, Remus runs completely transparently, requiring no changes to your existing guests.

This release works with the tip of the xen-unstable repository, and supports PV and HVM in 32-on-32, 64-on-64, 32-on-64, and 64-on-32 configurations. It has been tested using Linux (Ubuntu) PV guests, and both Linux and Windows XP under HVM.

I believe that it is now ready for inclusion in Xen, and I would love to have people try it out and let me know how it goes.

You can find more information, including installation and usage instructions, at the web site:


The web site includes a list of improvements I have in mind over the short term. Ideas (and patches) welcome!

Backport of igb driver to linux-2.6.18-xen

From xen-devel (Simon Horman):

I am happy to announce a refreshed backport of the IGB driver to linux-2.6.18-xen.

To allow the creation of virtual functions (VF) on the 82576 that can be passed through to domUs. This back-port is successful in that regard.

The current igb driver in linux-2.6.18-xen seems to be a back-port of a driver from RHEL5 that predates anything present in Linus’s kernel tree.

This makes back-porting of changes from Linus’s tree difficult. In particular, the original commit to Linus’s tree includes multi-queue support, a non-trivial feature, whereas the code currently in linux-2.6.18-xen does not.

With this in mind the method used was to back-port the original commit made to Linus’s tree and then back-port subsequent commits to Linus’s tree.

Although somewhat verbose this method appears to have worked quite well.

Some patches were not back-ported, in particular GRO. This is because the back-port seemed to be quite difficult for little gain. I’m happy to re-consider any non-back-ported patches if there is interest.

In total there are 191 patches on top of linux-2.6.18-xen

“xen/x86: make do_settimeofday() return -EPERM when clock can’t be changed”

(930:506133327116). Apart from one patch written by myself they all correspond to patches in Linus’s kernel tree. Where the patches have been modified I have noted this in the changelog and added my own Signed-off-by line. The majority of patches are unmodified.

Due to the large number of patches I am providing this backport as an hg tree, http://hg.vergenet.net/xen/linux-2.6.18-xen-igb

At this time it should be possible to pull this tree into linux-2.6.18-xen or to use it independently.

I am happy to post the patches to xen-devel if that is of value.

However my experience in the past has shown that posting more than a handful of patches results in a re-send-fest due to mangling issues.

There is also an issue that the first patch “igb: port to driver originally committed to Linus’s tree” (931:6b25a2daf6bc) is rather large at 192Kbytes.

Known Limitations

* This back-port does not include the VF driver (igbvf). This means that VFs created using the igb driver can’t be used by dom0. They can, however, be passed through to domUs.

* removing the igb module while a VF is passed-through results in a hypervisor panic. This is after several pciback messages which explain that you have done something that you sholdn’t.

pciback: ****** removing device 0000:02:10.1 while still in-use! ******

pciback: ****** driver domain may still access this device’s i/o resources!

pciback: ****** shutdown driver domain before binding device

pciback: ****** to other drivers or domains

(XEN) [VT-D]iommu.c:1271:d0 domain_context_unmap:PCIe: bdf = 2:10.1

(XEN) Xen BUG at msi.c:776

* The tree does not compile with PCI_IOV enabled between changesets

– “igb: this patch addes the sr-iov enablement option via num_vfs parameter”


– “igb: remove sysfs entry that was used to set the number of vfs”


This is because the first patch makes use of netdev->dev which does not exist. However the need for this is removed by the second patch.

Xen Development Tree (Linux Version) Update

The Xen Community has completed a discussion around the selection of the proper tree for future development activities;

from Keir Fraser on June 4th  (full thread)…

With 3.4 out the door it is time to revisit the state of our Linux repositories. Currently we have a number of trees in various states of maintenance: – linux-2.6.18-xen.hg: the ‘original’ tree. Still maintained, used and tested but increasingly long in the tooth. – ext/linux-2.6.27-xen.hg: a snapshot of opensuse’s kernel port. This clones tree is not maintained or tested. – XCI/linux-2.6.27.git: a forward port of the Xen patches to 2.6.27. Maintained as part of XCI project. – Jeremy’s pv_ops patches against kernel.org: maintained, (somewhat) tested, but incomplete.

It is probably time to kill the 2.6.18 tree, or at least stop active development within it. It is increasingly a kludged collection of backports of more recent kernel patches, and is also missing a lot of drivers for more modern hardware.

Our proposal is to move XCI’s linux-2.6.27 tree out of the XCI subproject and make it the main user tree. Development and automated testing would occur on that tree and of course on Jeremy’s pv_ops patchset (which we want to completely move onto at some point in the future).

The community has decided to move all development to “Jeremy’s pv_ops tree” as the new platform. Jeremy’s tree contains the 2.6.29 Linux kernel with associated patches and will end the Xen’s use of 2.6.18 on all future releases. The tree switch is anticipated to be complete by the middle of next week and I will post the information on this blog.

As I posted late last week, Xen is still engaged with the kernel.org team to move pv_ops Dom0 into upstream.