Monthly Archives: October 2012

The Paravirtualization Spectrum, Part 2: From poles to a spectrum

In part 1 of this series, I introduced the concepts of full virtualization and paravirtualization (PV), as well as the hardware virtualization (HVM) feature used by Xen (among other things) to implement full virtualization. I also introduced the concept of installing paravirtualized drivers on a fully virtualized system.

This small step, from full virtualization towards paravirtualization, begins to hint at the idea of a spectrum of paravirtualization. In this article, I will cover the historical reasons for the development of PVHVM, and finally of the newest mode, PVH.

Problems with paravirtualization: AMD and x86-64

It comes as a surprise to many people that while 32-bit paravirtualized guests in Xen are faster than 32-bit fully virtualized guests, when running in 64-bit mode, paravirtualized guests can sometimes be slower than fully virtualized guests. This is due to some changes AMD made when designing the architecture which simplified things for them, but made things more difficult for Xen.

Most modern operating systems need just two levels of protection: user mode and kernel mode. Kernel mode memory is protected from user mode memory via the pagetable “supervisor mode” bit.

Continue reading

Fedora 18 Virtualization Test Day

As usual, Fedora has planned a number of test days for their upcoming Fedora 18 release, including include a Virtualization Test Day on November 1st (tomorrow!).Fedora Logo

We are therefore calling all our community members to participate in the test day as much as possible. Specific information regarding testing Xen on the new Fedora can be found in this Wiki page. For attending and participating, be sure you hang out on IRC at #fedora-test-day (Freenode) on Thursday !

Fedora 18 will be one of the first distros shipping Xen 4.2… Join and help us making sure it will work great for all Fedora 18 future users !!

Results from the XCP User Survey

A few weeks ago, we launched our first XCP User Survey. We nearly had 500 responses when I compiled a preliminary analysis of the survey results. First, I wanted to thank everybody who participated. Of all respondents 38% of respondents are using XCP today, 40% are evaluating XCP or planning to do so and 22% just filled out the survey. Amongst people using or evaluating XCP, 56% of respondents are interested in XCP for personal purposess and 44% for business reasons. Interestingly enough, the proportion of XCP users amongst business users is 62% compared to 25% of personal users.

For, the remainder of this report we removed respondents who are neither using XCP nor planning to do so. Below are some interesting facts.

Users by Continent

Continue reading

The Paravirtualization Spectrum, part 1: The Ends of the Spectrum

At XenSummit 2012 in San Diego, Mukesh Rathor from Oracle presented his work on a new virtualization mode, called “PVH”. Adding this mode, there are now a rather dizzying array of different terms thrown about — “HVM”, “PV”, “PVHVM”, “PVH” — what do they all mean? And why do we have so many?

The reason we have all these terms is that virtualization is no longer binary; there is a spectrum of virtualization, and the different terms are different points along the spectrum. Part of the reason the terminology is a little unclear is the history; any language and terminology evolves over time in response to the changing situation. However, changing the terminology after-the-fact once certain usages become common is difficult.

So in this series of articles I will introduce just enough history to understand how the current situation came about, and (hopefully) introduce a consistent set of terminology which may help clear things up, while balancing this against the fact that people will still continue to use existing terminology.

This article will be divided into two parts. The first part will give a general introduction to virtualization, and to paravirtualization, Xen’s unique contribution to the field, as well as the advent of hardware virtualization extensions (HVM). It will also introduce the idea of adding paravirtualized drivers for disk and network. The second mode will cover the motivation and technical descriptions of two more modes which further mix elements of full virtualization and paravirtualization.

Full virtualization

In the early days of virtualization (at least in the x86 world), the assumption was that you needed your hypervisor to provide a virtual machine that was functionally nearly identical to a real machine. This included the following aspects:

  • Disk and network devices
  • Interrupts and timers
  • Emulated platform: motherboard, device buses, BIOS
  • “Legacy” boot: i.e., starting in 16-bit mode and bootstrapping up to 64-bit mode
  • Privileged instructions
  • Pagetables (memory access)

In the early days of x86 virtualization, all of this needed to be virtualized: disk and network devices needed to be emulated, as did interrupts and timers, the motherboard and PCI buses, and so on. Guests needed to start in 16-bit mode and run a BIOS which loaded the guest kernel, which (again) ran in 16-bit mode, then bootstrapped its way up to 32-bit mode, and possibly then to 64-bit mode. All privileged instructions executed by the guest kernel needed to be emulated somehow; and the pagetables needed to be emulated in software.

This is mode, where all of the aspects the virtual machine must be functionally identical to real hardware, is what I will call fully virtualized mode.

Xen and paravirtualization

Unfortunately, particularly for x86, virtualizing privileged instructions is very complicated. Many instructions for x86 behave differently in kernel and user mode without generating a trap, meaning that your options for running kernel code were to do full software emulation (incredibly slow) or binary translation (incredibly complicated, and still very slow).

The key question of the original Xen research project at Cambridge University was, “What if instead of trying to fool the guest kernel into thinking it’s running on real hardware, you just let the guest know that it was running in a virtual machine, and changed the interface you provide to make it easier to implement?” To answer that question, they started from the ground up designing a new interface designed for virtualization. Working together with researchers at both the Intel and Microsoft labs, they took both Linux and Windows XP, and ripped out anything that was slow or difficult to virtualize, replacing it with calls into the hypervisor (hypercalls) or other virtualization-friendly techniques. (The Windows XP port to Xen 1.0, as you might imagine, never left Microsoft Research; but it was benchmarked in the original paper.)

Continue reading on

One of the issues in any software project can be the disconnect between what users want or would find useful, and the developers’ idea of what users want or would find useful, and Xen is no exception.

To help address this issue, we are starting to experiment with a website called UserVoice. UserVoice allows users to do two things:

  • Suggest improvements or features
  • “Vote” for which improvements or features they think are most important.

You can find the uservoice page here:

You can either create an account, or log in with an existing Google or Facebook account.

Obviously we can’t promise that everything that is suggested or voted highly will be implemented! But hopefully it will give the developers a better idea what our users are thinking.

To make sure that the site is as useful as possible, please try focus on things you want to do, rather than how you want to be able to do them. That will help us choose the best way to help you; or perhaps point you to an existing way of doing something that’s already available.


Xen ARM in Linux!

Last weekend Linus Torvalds pulled the Xen on ARM patches in his Linux tree, so as of Saturday the 7th of October, we have Xen ARM in upstream Linux!

This makes Xen the first hypervisor supported by Linux on the ARM platform!

Working on ARM has been a very pleasant experience for me: the documentation of the hardware is well written and complete, the virtualization extensions are useful and fit our architecture very well, the ARM emulator comes with a nice debugger that helped me figure out some of the most difficult problems I had.

But beyond the hardware and the development tools, I was really impressed by how welcoming the Linux ARM Community has been to me: from the invite to the ARM session at the Kernel Summit, to the guidance through the upstreaming process and in general the feedback I received to my work. Nowadays, the Linux Community is usually perceived as not being friendly to newcomers, but it hasn’t certainly been the case for me! In particular I would like to thanks Arnd Bergmann, Marc Zyngier, and the Linaro folks. It has been great working with you. I would also like to thank Konrad Rzeszutek Wilk because he went out of his way to help me upstream my ARM work: I couldn’t have done it without him.

As the Xen on ARM patches were pulled by Linus, I was also appointed as Xen ARM maintainer in the Linux kernel. This is a new responsibility for me and I am not going to take it lightly. I am looking forward to work closely with Arnd, Russell, Konrad and the other Linux maintainers to make Linux the best operating system to run virtual machines and inside virtual machines on Xen on ARM!

What’s next

While Xen on ARM in Linux is certainly a major milestone, there are still a lot of things to do. Right now I am busy trying to run Xen on a Cortex A15 Versatile Express development platform while Ian Campbell already started the ARMv8 Xen port!
Stay tuned if you want to run Xen 64-bit on ARM.

See Also

If you are interested in the Xen on ARM project, you might want to read the slides of the presentation I gave at XenSummit 2012: