Monthly Archives: January 2014

Linux 3.14 and PVH

The Linux v3.14 will sport a new mode in which the Linux kernel can run thanks to Mukesh Rathor (Oracle).

Called ‘ParaVirtualized Hardware,’ it allows the guest to utilize many hardware features – while at the same time having no emulated devices. It is the next step in PV evolution, and it is pretty fantastic.

Here is a great blog that explains the background and history in detail at:
The Paravirtualization Spectrum, Part 2: From poles to a spectrum.

The short description is that Xen guests can run as HVM or PV. PV is a mode where the kernel lets the hypervisor program page-tables, segments, etc. With EPT/NPT capabilities in current processors, the overhead of doing this in an HVM (Hardware Virtual Machine) container is much lower than the hypervisor doing it for us. In short, we let a PV guest run without doing page-table, segment, syscall, etc updates through the hypervisor – instead it is all done within the guest container.

It is a hybrid PV – hence the ‘PVH’ name – a PV guest within an HVM container.
Continue reading

Improved Xen support in FreeBSD

FreeBSD Logo
As most FreeBSD users already know, FreeBSD 10 has just been released, and we expect this to be a very good release regarding Xen support. FreeBSD with Xen support includes many improvements, including several performance and stability enhancements that we expect will greatly please and interest users. With many bug fixes already completed, the following description only focuses on new features.

New vector callback

Previous releases of FreeBSD used an IRQ interrupt as the callback mechanism for Xen event channels. While it’s easier to setup, using a IRQ interrupt doesn’t allow to inject events to specific CPUs, basically limiting the use of event channels in disk and network drivers. Also, all interrupts were delivered to a single CPU (CPU#0), not allowing proper interrupt balancing between CPUs.

With the introduction of the vector callback, events can now be delivered to any CPU, allowing FreeBSD to have specific per-CPU interrupts for PV timers and PV IPIs, and balancing the others across the several CPU usually available on a domain.

PV timers

Thanks to the introduction of the vector callback, now we can make use of the Xen PV timer, which is implemented as a per-CPU singleshot timer. This alone doesn’t seem like a great benefit, but it allows FreeBSD to avoid making use of the emulated timers, greatly reducing the emulation overhead and the cost of unnecessary VMEXITs.


As with PV timers, the introduction of the vector callback allows FreeBSD to get rid of the bare metal IPI implementation, and instead route IPIs through event channels. Again, this allows us to get rid of the emulation overhead and unnecessary VMEXITS, providing better performance.

PV disk devices

FLUSH/BARRIER support has been recently added, together with a couple of fixes that allow FreeBSD to run with a CDROM driver under XenServer (which was quite of a pain for XenServer users).

Support for migration

With these new features, migration doesn’t break since it has been reworked to handle the fact that timers and IPIs are also paravirtualized now.

Merge of the XENHVM config into GENERIC

One of the most interesting improvements from a user/admin point of view (and something similar to what the pvops Linux kernel is already doing), the GENERIC kernel on i386 and amd64 now includes full Xen PVHVM support, so there’s no need to recompile a Xen-specific kernel. When run as a Xen guest, the kernel will detect the available Xen features and automatically make use of them in order to obtain the best possible performance.

This work has been done in conjunction between Spectra Logic and Citrix.

libvirt support for Xen’s new libxenlight toolstack

Originally posted on my blog, here.

Xen has had a long history in libvirt.  In fact, it was the first hypervisor supported by libvirt.  I’ve witnessed an incredible evolution of libvirt over the years and now not only does it support managing many hypervisors such as Xen, KVM/QEMU, LXC, VirtualBox, hyper-v, ESX, etc., but it also supports managing a wide range of host subsystems used in a virtualized environment such as storage pools and volumes, networks, network interfaces, etc.  It has really become the swiss army knife of virtualization management on Linux, and Xen has been along for the entire ride.

libvirt supports multiple hypervisors via a hypervisor driver interface, which is defined in $LIBVIRT_ROOT/src/drvier.h – see struct _virDriver.  libvirt’s virDomain* APIs map to functions in the hypervisor driver interface, which are implemented by the various hypervisor drivers.  The drivers are located under $LIBVIRT_ROOT/src/<hypervisor-name>.  Typically, each driver has a $LIBVIRT_ROOT/src/<hypervisor-name>/<hypervisor-name>_driver.c file which defines a static instance of virDriver and fills in the functions it implements.  As an example, see the definition of libxlDriver in $libvirt_root/src/libxl/libxl_driver.c, the firsh few lines of which are

static virDriver libxlDriver = {
    .no = VIR_DRV_LIBXL,
    .name = “xenlight”,
    .connectOpen = libxlConnectOpen, /* 0.9.0 */
    .connectClose = libxlConnectClose, /* 0.9.0 */
    .connectGetType = libxlConnectGetType, /* 0.9.0 */

Continue reading

First Xen Project 4.4 Test Day on Monday, January 20

Release time is approaching, so Xen Project Test Days have arrived!

On Monday, January 20, we are holding a Test Day for Xen 4.4. Release Candidate 2.

Xen Project Test Day is your opportunity to work with code which is targeted for the next release, ensure that new features work well, and verify that the new code can be integrated successfully into your environment.  This is the first of a few Test Days for the 4.4 release, scheduled to occur at roughly 2 week intervals.

General Information about Test Days can be found here:

and specific instructions for this Test Day are located here:


If you have a new feature which is cooked and ready for testing in RC2, we need to know about it and how to test it. Either edit the instructions page or send me a few lines describing the feature and how it should be tested.

Right now, RC2 is labelled a general test (e.g., “Does Xen compile, install, and do the things Xen normally does?”). We don’t have any specific tests of new functionality identified. If you have something new which needs testing in RC2, we need to know about it.


Please join us on Monday, January 20, and help make sure the next release of Xen is the best one yet!

Xen Related Talks @ FOSDEM 2014

Going to FOSDEM’14? Well, you want to check out the schedule of the Virtualization & IaaS devroom then, and make sure you do not miss the talks about Xen. There are 4 of them, and they will provide some details about new and interesting usecases for virtualization, like in embedded systems of various kind (from phones and tablets to network middleboxes), and about new features in the upcoming Xen release, such as PVH, and how to use them with profit.

Here they are the talks, in some more details:
Dual-Android on Nexus 10 using XEN, on Saturday morning
High Performance Network Function Virtualization with ClickOS, on Saturday afternoon
Virtualization in Android based and embedded systems, on Sunday morning
How we ported FreeBSD to PVH, on Sunday afternoon

There actually is more: one called Porting FreeBSD on Xen on ARM, in the BSD devroom, and one about MirageOS one in the miscellaneous Main track, but the schedule for them has not been announced yet.

Last but certainly not least, there will be a Xen-Project booth, where you can meet the members of the Xen community as well as enjoying some other, soon to be revealed, activities. I and some of my colleagues from Citrix will be in Brussels, and will definitely spend some time at the booth, so come and visit us. The booth will be in building K, on level 1.

Read more here:


The schedule for the FreeBSD and MirageOS talks have been announced. Here it comes:
Porting FreeBSD on Xen on ARM, will be given on Saturday early afternoon (15:00), in the BSD devroom
MirageOS: compiling functional library operating systems, will happen on Sunday late morning (13:00), in the misc main track

Also, there is another Xen related talk, in the Automotive development devroom: Xen on ARM: Virtualization for the Automotive industry, on Sunday morning (11:45).