Author Archives: Ian Campbell

About Ian Campbell

Ian Campbell has been involved with the Xen project since joining XenSource in 2005. Today he is a Principal Software Engineer at Citrix Systems, Inc ( working on Xen where his interests include Xen on ARM, Linux on Xen, paravirtualised networking and toolstack issues. Prior to Citrix (and XenSource) he worked on embedded Linux systems at Arcom Control Systems. Ian is a committer, Linux maintainer (Xen network backend driver) and Debian Developer. Also check out Ian's personal blog.

Using Grub 2 as a bootloader for Xen PV guests

Background: Introduction to Xen PV Bootloaders

In the very early days of Xen it was necessary for the host (domain 0)
administrator to explicitly supply a kernel (and perhaps initial
ramdisk) from the domain 0 filesystem in order to start a new guest.

This mostly worked and for some use cases, i.e. those where the host
admin wants very strict control over what each guest runs, was
desirable and remains so today.

However for other use cases it was rather inflexible since it meant
that the host administrator needed to be involved in what many
considered to be a guest administrator, or even distribution level,
decision i.e. the selection of which kernel to run, with what
parameters etc. Continue reading

Xen on ARM and the Device Tree vs. ACPI debate

ACPI vs. Device Tree on ARM

Some of you may have seen the recent discussions on the linux-arm-kernel mailing list (and others) about the use of ACPI vs DT on the ARM platform. As always LWN have a pretty good summary (currently subscribers only, becomes freely available on 5 December) of the situation with ACPI on ARM.

Device Tree (or DT) and Advanced Configuration & Power Interface (or ACPI) are both standards which are used for describing a hardware platform e.g. to an operating system kernel. At their core both technologies provide a tree like data structure containing a hierarchy of devices and specifying what type they are and a set of “bindings” for that device. A binding is essentially a schema for specifying I/O regions, interrupt mappings, GPIOs and clocks etc.

For the last few years Linux on ARM has been moving away from hardcoded “board files” (a bunch of C code for each platform) towards using Device Tree instead. In the ARM space ACPI is the new kid on the block and has many unknowns. Given this the approach to ACPI which appears to have been reached by the Linux kernel maintainers, which is essentially to wait and see how the market pans out, seems sensible.

On the Xen side we started the port to ARM around the time that Linux’s transition from board files to Device Tree was starting and made the decision early on to go directly to device tree (ACPI wasn’t even on the table at this point, at least not publicly). Xen DT to discover all of the hardware on the system, both that which it intends to use itself and that which it intends to pass to domain 0. As well as consuming DT itself Xen also creates a filleted version of the host DT which it passes to the domain 0 kernel. DT is simple and yet powerful enough to allow us to do this relatively easily.

DT is also used by some of the BSD variants in their ARM ports as well.

My Position as Xen on ARM Maintainer

The platform configuration mechanism supported by Xen on ARM today is Device Tree. Device Tree is a good fit for our requirements and we will continue to support it as our primary hardware description mechanism.

Given that a number of operating system vendors and hardware vendors care about ACPI on ARM and are pushing hard for it, especially in the ARM server space, it is possible, perhaps even likely, that we will eventually find ourselves needing support ACPI as well. On systems which support both ACPI and DT we will continue to prefer Device Tree. Once ARM hardware platforms that only support ACPI are available, we will obviously need to support ACPI.

The Xen Project works closely with the Linux kernel and other open source upstreams as well as organisations such as Linaro. Before Xen on ARM can support ACPI I would like see it gaining some actual traction on ARM. In particular I would like to see it get to the point where it has been accepted by the Linux kernel maintainers. It is clearly not wise for Xen to be pioneering the use of ACPI before to it becoming clear whether or not it is going to gain any traction in the wider ecosystem.

So if you are an ARM silicon or platform vendor and you care about virtualization and Xen in particular, I encourage you to provide a complete device tree for your platform.

Note that this only applies to Xen on ARM. I cannot speak for Xen on x86 but I think it is pretty clear that it will continue to support ACPI so long as it remains the dominant hardware description on that platform.

It should also be noted that ACPI on ARM is primarily a server space thing at this stage. Of course Xen and Linux are not just about servers: both communities have sizable communities of embedded vendors (on the Xen side we had several interesting presentations at the recent Xen Developer Summit on embedded uses of Xen on ARM). Essentially no one is suggesting that the embedded use cases should move from DT to ACPI and so, irrespective of what happens with ACPI, DT has a strong future on ARM.

ACPI and Type I Hypervisors

Our experience on x86 has shown that the ACPI model is not a good fit for Type I hypervisors such as Xen, and the same is true on ARM. ACPI essentially enforces a model where the hypervisor, the kernel, the OSPM (the ACPI term for the bit of an OS which speaks ACPI) and the device drivers all must reside in the same privileged entity. In other words it effectively mandates a single monolithic entity which controls everything about the system. This obviously precludes such things as dividing hardware into that which is owned and controlled by the hypervisor and that which is owned and controlled by a virtual machine such as dom0. This impedance mismatch is probably not insurmountable but experience with ACPI on x86 Xen suggests that the resulting architecture is not going to be very agreeable.


Due to their history on x86 ACPI and UEFI are often lumped together as a single thing when in reality they are mostly independent. There is no reason why UEFI cannot also be used with Device Tree. We would expect Xen to support UEFI sooner rather than later.

Reporting A Bug Against the Xen Hypervisor

With the release process for Xen 4.3 in full swing (we intend to release the third release candidate this week) and with the Xen Test Days initiative (the next one is this Wednesday 5 June, join us on IRC freenode #xentest) I thought it would be useful to offer up some information and guidance on how the Xen project deals with bug reports and how to report bugs against the Xen Hypervisor Project.

Reporting a Bug

Confirm That Your Issue Is a Bug

Experience has shown that oftentimes what appears to be a bug is often just a misconfiguration or misunderstanding of how things are supposed to work. This can be down to a lack of documentation (a perennial problem for Open Source projects and something which we are working to address with our regular Xen Document Days) or the inherent complexity of something of the things which one can achieve (or try to achieve!) using Xen.

With that in mind it is often useful to try seeking help via other means before resorting to submitting a bug report. Useful resources for asking questions are:

  • In a Xen system logs the logs can usually be found in /var/log/xen and will sometimes provide useful insight into an issue.
  • Search engines. As well as simply searching for key terms relating to your issue you can also search the User and Developer list archives.
  • Continue reading

Using valgrind to debug Xen toolstacks

What is Valgrind?

Valgrind is a framework for building dynamic analysis tools. Several useful tools are included in the Valgrind distribution including tools to check dynamic memory usage (memcheck), a cache profiler (cachegrind), heap profiler (massif) and thread debugger (helgrind) among others. Valgrind also provides a framework which can be used to build other tools.

The Valgrind tool which I find most useful and the one which I have most experience with is memcheck. This tool can detect all manner of memory management problems, including use after free, using uninitialized data, memory leaks, double free. Between them these can result in savings of many hours of staring a core dumps and gdb backtraces.

How Does memcheck Work?

At its core Valgrind is a dynamic binary translation engine, which is used to instrument the code at runtime. In the case of memcheck this is used to track properties of the memory in a process, including aspects such as whether each bit (yes, bit) of memory has been initialized since it was allocated. It also tracks this information as data is loaded into registers, so it can know if a given register is currently tainted with uninitialized data or not. As well as instrumentation through binary translation Valgrind also includes instrumented versions of the C dynamic memory allocation functions which are used to track whether a each bit of memory is currently considered free or allocated, as well as tainting registers when they contain a pointer to memory which has been freed.

Continue reading

Xen 4.2.0 Released is pleased to announce the release of Xen 4.2.0. The release is available from the download page:

This release is the culmination of 18 months and almost 2900 commits and almost 300K lines of code of development effort, by 124 individuals from 43 organizations.

New Features

The release incorporates many new features and improvements to existing features. There are improvements across the board including to Security, Scalability, Performance and Documentation.

XL is now the default toolstack: Significant effort has gone in to the XL tool toolstack in this release and it is now feature complete and robust enough that we have made it the default. This toolstack can now replace xend in the majority of deployments, see XL vs Xend Feature Comparison. As well as improving XL the underlying libxl library has been significantly improved and supports the majority of the most common toolstack features. In addition the API has been declared stable which should make it even easier for external toolstack such as libvirt and XCP’s xapi to make full use of this functionality in the future.

Large Systems: Following on from the improvements made in 4.1 Xen now supports even larger systems, with up to 4095 host CPUs and up to 512 guest CPUs. In addition toolstack feature like the ability to automatically create a CPUPOOL per NUMA node and more intelligent placement of guest VCPUs on NUMA nodes have further improved the Xen experience on large systems. Other new features, such as multiple PCI segment support have also made a positive impact on such systems.

Continue reading

Xen 4.2 Release

Just a quick update on the Xen 4.2 release. 4.2.0-rc3 was released on 23 August and we intend to release rc4 around the end of the week. Barring any last minute critical bug reports our intention is that rc4 will be the final release candidate.

  • 19 March — TODO list locked down
  • 2 April — Feature Freeze
  • 30 July — Xen 4.2.0-rc1
  • 8 August — Xen 4.2.0-rc2
  • 23 August — Xen 4.2.0-rc3
  • 6/7 September — Xen 4.2.0-rc4 PLANNED

Looking forward to 4.3 George Dunlap has announced that he intends to help coordinate the release. If you intend to work on a feature for the 4.3 release please let George know.