Author Archives: Anthony PERARD

QEMU vs. qemu-traditional update

Here is an update about feature completeness of QEMU compared to the old qemu-traditional.

But first, what is the difference between QEMU and qemu-traditional?

QEMU is the software that can be found at, we can also call it QEMU upstream. It’s where all new features are supposed to land.

What we call “qemu-traditional” is the fork of QEMU that has been used by Xen. It became harder and harder to maintain and to upgrade with recent version QEMU, so we could not benefit from some of the new features that have been developed in QEMU, and also any bug that we would have found in the fork can be hard to fix upstream because the code would be very different.

So, we took everything that was needed from qemu-traditional to run QEMU with Xen and integrate them in a modern QEMU. Up to now, few features were missing to be able to use QEMU, but now, all the main features are in and QEMU became the default for most usage.

So what’s new in 4.3?

An important feature to be able to live-migrate a guest is a way to be able to track memory that has been modified during a live-migration. The feature first appears in Xen 4.2.2.

We also added the support to CPU hotplug for HVM as it was one of the missing features to get QEMU closer to qemu-traditional and have it as default.

Missing feature?

We are still missing a few things with QEMU upstream, so far, the limited support for VGA passthrough has not been upstream. Another missing feature would be the use of QEMU in a stub-domain instead of using qemu-traditional as it is right now. This last one is planned to be fixed in hopefully 4.4.

There is also patchs to fix the suspend resume cycle of an HVM guest that should be applied soon.

Beside those missing features, there are more works going one to enhance the support of many QEMU features that are not usable right now with the tool stack libXL, like using USB redirection or USB passthrough or even one day supporting QXL.

Bringing Xen on the Chromebook and the Arndale Board: the journey so far

I’ve started to work on the Xen on ARM project by trying to get Xen running on the nice little Samsung Chromebook which run an ARM processor with the virtualization extensions. The Chromebook uses Exynos 5250 dual core platform with 2GB of RAM, which could be perfect for a good demo of Xen on ARM.

But porting Xen to new ARM platform without any useful debug output of some kind was really hard at the beginning. There is no accessible serial port available on the Chromebook and Xen cannot display anything on the screen. But there is a trick, write your log to specific area in RAM, and read it back after a reboot. It is really annoying, does take more time to have any feedback from Xen and you don’t know when you can reboot the machine to get those logs. This is how we did it, until we received an Arndale Development board.

So, we started the development on Chromebook by simply installing a Linux on it (on an SD card). There were few tutorial on internet, so it was not too hard. Then we recompiled a kernel, as it will be needed to integrate dom0 support. And last, we tried to understand how to use the bootloader on the machine to load Xen where we need it to be and execute it, it took some time and many tries. When we finally get something from Xen, we could continue make it work on the Chromebook. We had bugs like loading Xen at the wrong place in RAM or missing information in the device tree. Then it took sometime to load Linux correctly and we had to use some hack to actually find the Linux blob in memory as the default bootloader was not very helpful for that.

We found several problems were a page table was written in the middle of the binary in memory because of too many hack on how to load Linux for dom0. Before starting to work with the Arndale, we’ve been able to run some of Linux code, but fail early in Linux when it tried to load the device tree.

The Arndale board finally arrived, we had to start over, different bootloader and this time there is a serial port!! The Arndale use the same hardware platform, namely, the Exynos 5.

So first thing, how to start Xen on it? It appears to be relatively easy to load your own kernel, Xen here. So let’s hack a serial driver for Xen, in assembly first :-). We found out that the bootloader delivered with the board does not let Xen switch to hypervisor mode, so we had to recompile a u-boot for the board. Not a big deal, Linaro has done some work on the board as well, so we could get u-boot from their source tree and finally have Xen get its hypervisor mode.

After the first bits resolved, we could get along and figured out all the hack/fix needed in Xen in order to run it on this second platform, the first one was a Versatile Express. There were a lot of assumption about the hardware. So to get things running, we just replace those by others because having a proper way to handle different platform take a lot of time.

We had an issue with the time on this platform. The ARM Generic Timer was not working, it was always giving the same time. After trying to use the Multi-Core Timer of the Exynos platform, we realise there was the same, but we had to set a bit in the MCT register in order for the timer to start counting.

After resolving few other issues, we could finally get a prompt! Next, we worked on getting more, like starting few guest.

There is yet another hack for both the Arndale and the Chromebook to get a working network interface or having something on the screen of the Chromebook, it’s a 1:1 mapping of the memory of dom0. This is when the virtual address (seen by dom0) match the physical address (seen by the hardware). Normaly, those two addresses are different for any guest.

To conclude, it has been a lot of fun working on those two. We could get a working demo for both the Arndale and the Chromebook of Xen capabilities. There is still a lot of work ahead of us, but we are getting there.

Much more happened in the latest weeks, during which I concentrated on something different than Xen on ARM or the Chromebook. For instance, there is now a better multi-platform support in Xen and SMP support is getting there as well.

Here is some wiki page on how to get started:

Linux Stub-Domain

The “Normal” Case

To explain what is a Stub-Domain (often called stubdom), let’s start with the basics. When you start a new guest with Xen, you would need a Device Model which does some emulation if the guest does not have PV drivers. This is the case for an HVM domain.
This device model (which is QEMU), needs to run somewhere. By default it runs in the Dom0, as root.

Here is a picture:

Now, every time the guest do an IO, like reading a file on the disk, there is an event sends through Xen to the device model. It does the emulation and send the result to the guest.

Continue reading

PCI Passthrough in QEMU

We now have PCI passthrough support in QEMU upstream, this was one of the missing pieces needed to have a full featured QEMU device model. But there is still more work to do on it.

Why do we use QEMU?

We use QEMU in Xen to emulate a part of the hardware, in particular a disk, a network card and the graphic output. So a guest does not need to be modified in order to run under Xen. Even with a modified guest, QEMU can be used to handle the graphic output of the guest, or as a backend for a paravirtualized disk or network card.

A blog post a year ago mentions that we had Xen support in QEMU, this was the first part needed to run a Xen guest with QEMU upstream. Now, with the PCI passthrough support upstreamed, that one large piece of code closer to a full featured QEMU upstream.

What is PCI passthrough?

Also known as device assignment, PCI passthrough is here to assign a real PCI device to a guest, like a network card or a sound card. This give full and direct access to the PCI device from a fully virtualized guest.

To use it, take a look at our wiki: PCI passthrough with Xen 4.2 as there is no change compared to the traditional qemu-dm. You can take a look here as well: Xen PCI Passthrough for more general information about PCI passthrough.

Continue reading

Xen support upstreamed to QEMU

My name is Anthony Perard and I have been working on up-streaming Xen patches to the QEMU mainline. The good news is that the bulk of the changes for Xen are in the QEMU mainline as of yesterday.

Let me begin a year ago, when I started working on this project. The Xen codeline contained a fork of the QEMU code implementing a “xen” target within QEMU.  The drawback of using a “xen” specific target within the QEMU source tree, was that lots of QEMU code had to be duplicated, in effect increasing the maintenance burden on the QEMU project. Unsurprisingly the code as it was, wasn’t acceptable.

Thus we embarked on a project to rewrite the Xen QEMU code. The goal was to add Xen support to QEMU with the goal to avoid code duplication.  It has taken some time: but after several revisions of the patch series, the code has finally been accepted into the QEMU mainline. A lot of effort has been spent on ensuring the quality of the changes are very high straight from the outset. This is not the end of the road: some additional patches for PCI pass-through and support for VGA dirty bitmaps are still outstanding.

Xen support in QEMU will have advantages for the QEMU as well as the Xen community. First of all, the last year has brought the two communities closer together. Going forward we hope that there will be more collaboration and sharing of ideas. Of course for the Xen community there are also advantages using vanilla QEMU: we will have access to more functionality as it is developed, such as support for the Q35 chipset within QEMU, PCI express support, and others.

I am looking forward to working more on this in the future and also wanted to thank Alexander Graf for feedback and suggestions on QEMU patches.