Monthly Archives: November 2011

Baremetal vs. Xen vs. KVM — Redux

The Xen community was very interested in (and a little worried by!) the recent performance comparison of ”Baremetal, Virtual Box, KVM and Xen”, published by Phoronix, so I took it upon myself to find out what was going on.

Upon investigation I found that the 3.0 Linux kernel used in Ubuntu 11.04 was lacking a rather key set of patches in domain 0 which inform the Xen hypervisor about the power management (specifically cpufreq scaling) properties of the processors in the system. Without these patches Xen will not make use of the highest performing CPU frequencies. These patches are in the process of being upstreamed to Linux but are already readily available and reasonably easy to apply to a 3.0 onwards kernel. You can find them at:

I reran the benchmarks presented by Phoronix in the following scenarios:

  • Baremetal Baseline
  • KVM Baseline
  • Xen PVHVM Baseline
  • Xen PVHVM Rebuilt
  • Xen PVHVM CPUFreq
  • Xen PVHVM 3.1+CPUFreq

The “Baseline” results are stock Ubuntu 11.04. “Xen PVHVM Rebuilt” is a straight rebuild of the stock Ubuntu 11.04 kernel (to rule out a simple rebuild impacting the results too much), “Xen PVHVM CPUFreq” is that stock kernel plus the cpufreq patches and “Xen PVHVM 3.1 +CPUFreq” is a mainline 3.1 plus those patches (only really included because that’s
where those patches were originally developed, comparing 3.0 and 3.1 is a bit apples and oranges). In all cases only the dom0 kernel was modified and the guest was always using the stock 11.04 kernel.

All test cases were run on the same hardware. The baremetal results used 32GB of RAM, 250GB disk and 16 cores while in all cases the virtual machines were given 24GB of RAM, 24GB of disk and 16 cores.

The Xen guest was using a “PVHVM” configuration, that is an HVM (fully-virtualised) guest making full use of paravirtualised drivers and PV extensions (PV timers, PV interrupt injection, all of which are enabled by default). The KVM guest was configured to use the virtio drivers for IO as well as any other paravirtualisation which is enabled by default.

Here are the raw results as reported by the Phoronix Test Suite:

The following table compares the baseline KVM figures (nb: the patches are to Xen specific code and will not impact KVM) to the “Xen PVHVM CPUFreq” case and tells a very different story to the numbers shown by Phoronix.

As you can see in many cases the results were very close (9/17 cases were +/- 1% in their respective comparison to native) and in the remaining 8 cases 4 favoured Xen and 4 KVM. Overall 7 cases favoured Xen and 8 favoured KVM with 2 having identical results. This is not surprising since many of the test cases are heavily CPU bound and you would therefore naturally expect that two virtualisation solutions making full use of hardware virtualisation facilities would be approximately equivalent.

I sent the above results and analysis to Michael Larabel, the Phoronix author of the article, on 17 November but have yet to hear any response. In the meantime he has posted another article containing results of a set of tests clearly chosen to highlight the power management impact of not applying these patches. It’s disappointing that Phoronix chose not to engage with the Xen community before publishing these results despite being contacted several times by a variety of people. Of course, we are not the only community which recently has been affected by unbalanced reporting (see “About the Kernel 3.0 “Power Regression” Myth”) and one would do well to think carefully about the reliability of performance measurements from folks who do not take minimal steps to understand or explain the results which they are seeing.

The full test results are available upon request. I won’t delve any deeper here since I don’t feel the kind of vacuous “analysis” performed by Phoronix really adds much to the raw data and there really isn’t much else to say about them.

In Summary

  • The results published by Phoronix in ”Baremetal, Virtual Box, KVM and Xen” which favour KVM over Xen are caused by missing patches in the Linux Kernel.
  • Patches which fix this issue are available. With those patches applied to the Dom0 kernel the performance measured using the Phoronix benchmarks are very similar on KVM and Xen.
  • The behavior observed by Phoronix mainly applies to hardware which aggressively uses the power management capabilities of processors (i.e. Laptops are more affected than servers).

Xen Document Day: November 29th

After the last Xen Document Day, many people asked me when we would do another document day. After a discussion on the mailing list, we settled on next Tuesday (November 29). This will be an all-day on-line IRC event with the aim to

  • Improve user documentation
  • Improve developer documentation, including the creation of man pages, etc.

There is no hard start and end to the event, but volunteers in the UK will be on-line on Nov 29th from around 9:00 UTC+1 until around 18:00 UTC+1, and then volunteers from the US will take over. The event will be take place on freenode channel #xendocday

An intial work item lists and what we agreed on can be found on the Documentation Day Etherpad. Other work that needs doing is:

But anyway, a long list of stuff that is needed can be found on the Documentation Day Etherpad. I am sure, you will find something that suits you. Instructions on how to use the new wiki, can be found here and on the front-page.

The Protocol

  • Join us on IRC: freenode channel #xendocday
  • Tell people what you intend to work on (to avoid doing something somebody else is already working on)
  • Fix some documentation
  • Help others
  • And above all: have fun!

I am looking forward to the day and hopefully documentation days will become a regular Xen event. See you on IRC!

Tommorrow: Cloud Discovery Days @ LinuxCon Brazil

A quick reminder that is helping to run the Cloud Discovery Days @ LinuxCon Brazil, tommrrow and Friday. We are doing this in partnership with Citrix, Dell and Rackspace.

For users of Xen, a number of interesting things will happen:

  • I will give a status update on PVOPS, the Xen Hypervisor project, XCP and Xen ARM: sharing the latest and greatest from XenSummit Asia, two weeks ago.
  • Marcus Granado will give a hands-on XCP session. He will cover some use-cases such as the one described in his blog post last week.
  • Boriz Quiroz will discuss a case study of an Asian company that, using cloud computing platform such as OpenStack and XCP has developed a comprehensive service network to e-learning.
  • Sheng Liang will share lessons on Open Source powered IaaS clouds
  • A CD with an XCP 1.1 image will be given to all attendees at the event and distributed via the Brazilian Linux Magazine. A big thank you to for making this happen.

There will also be many talks about Openstack and Cloud in general. Do check out the Cloud Discovery Days program page for more information.

Moving Xen Sites to Rackspace

Rackspace Hosting has offered the community to host services. We have started migration last week and the expectation is that all services will be migrated by Nov the 21st. Most services will be migrated with little impact on our users.

Where there is an impact, there will be an announcement on the Xen mailing lists. Some services, such as have already been migrated.

New Wiki

We have a new MediaWiki instance at which is currently in testing. We will be retiring the old MoinMoin Wiki and make it read-only on Monday Nov 14th at 12:00 GMT. At this point, we will rename to For now, you can check out the new Wiki, create accounts, etc. Note that only some Wiki pages have been migrated so far, but all the pages which have been marked as important by the community will be migrated by Monday.

The expectation is that

  • Pages marked as “Keep” in this spreadsheet will be migrated by Monday
  • Pages marked as “Update”, “Merge” and “Unknown” will be migrated later
  • Pages marked as “Archive” will be migrated last
  • All other pages will not be migrated

However, you will be able to find

We intentionally do not want to wholesale migrate all content as so much is out-of-date. The move to the new Wiki will provide an oppurtunity for a fresh start.

We will be retiring support for LXR, because LXR has been removed from Debian Squeeze and we do have source browsing capabilities on the repository pages.

Other changes
Some services such as mailing lists and the wiki will be migrated, but we will not initially reskin them. In other words: you will find that the top-level menus and footers of the sites as well as the styling will be different. We will fix these when all the services have been migrated. If you find any other issues, please send an e-mail to

XCP, RBAC and PAM authentication in the XenAPI

Many users have been asking a bit about the internals of the authentication mechanisms in XCP and XAPI, and how to add extra users, so I thought I should talk a bit about it.

Just after installation, XCP allows only root to log in, since by default this is the only user available over PAM/NSS in dom0. Extra users can be added locally to /etc/passwd or externally by using NSS/PAM extensions such as LDAP [1], NIS, winbind, Likewise etc in dom0. However, due to backwards-compatible behavior in the XenAPI that goes back to XenServer 5.0, all these users will still have root access by default, and therefore they will all be allowed access to all of the XenAPI calls.

In order to restrict the XenAPI calls for these extra users, it is necessary to make XAPI aware that an external authentication mechanism such as PAM is being used instead of the default backwards-compatible internal XAPI authentication. This can be accomplished by the following xe cli command, which will reroute all the XenAPI authentication requests to the external PAM XAPI plugin:

# xe pool-enable-external-auth auth-type=PAM service-name=<any-name>

A nice side-effect of enabling external authentication in XAPI as above is that all the XenAPI authentication requests will go through XAPI’s role-based access control (RBAC) mechanism. The RBAC in XAPI allows fine-tuning the permissions of any extra users or groups that are available over external authentication.

In XCP parlance, external users and groups are called “subjects”. By default all subjects will be denied access to the XenAPI. In order to allow any of them access to a subset (or the full set) of the XenAPI calls, two steps are necessary:

1. Add a subject to the subject-list in the XAPI database:

# xe subject-add subject-name=<external-user-or-group-name>

By default, a new subject will not be allowed to use any XenAPI calls.

2. Assign a set of XenAPI calls to the subject:

# xe subject-role-add role-name=<role-or-permission-name>

The pre-defined role names are pool-admin, pool-operator, vm-power-admin, vm-admin, vm-operator, read-only, which define a total order of permission sets ranging from all XenAPI calls (pool-admin) to only read-only calls (read-only). A relatively up-to-date table relating the available roles to permissions can be found at

Roles are only alias to a set of XenAPI calls. It is also possible to assign more than one XenAPI call to a subject. Therefore, it is possible to tweak the pre-defined roles by adding extra calls to it. For instance, to create a vm-operator that can also migrate VMs (whose XenAPI call is by default only available to pool-operator), you can do the following, assuming you have a subject called “migrate-op”:

# xe subject-role-add role-name=vm-operator uuid=<uuid-of-subject-"migrate-op">
# xe subject-role-add role-name=vm.pool_migrate uuid=<uuid-of-subject-"migrate-op">

You can then check what roles are assigned to what subjects:

# xe subject-list

The user root is still available for disaster-recovery purposes, in case either the external authentication mechanism stops working due to some problem in the external authentication plugin or the pool master’s subject list is not available due to some accident. In this case, slave emergency commands to recover the host or the pool can still use the user root.

Also, an interesting detail is that even though the XenAPI calls are usually available through the master only, you can call the XenAPI session.login_with_password(username,pwd) on any slave so that you can distribute the authentication load through all the hosts of the pool.

It is possible to list the available roles using the xe cli command

# xe role-list

but this will result in a simplified list of the default role names. In order to obtain the full list of all the available role names that can be added to a subject, you need to use the XenAPI directly, for instance using python:

$ python
>>> import xmlrpclib
>>> x=xmlrpclib.Server("https://my-xenserver-host")
>>> s=x.session.login_with_password("root","my-password")['Value']
>>> roles = x.role.get_all_records(s)['Value']
>>> print roles
{'Status': 'Success', 'Value': {'OpaqueRef:d7ea15b2-0664-1cd5-dc30-7c49a546ae80': {'subroles': [], 'name_description': 'A basic permission', 'name_label': 'pool.get_wlb_enabled', 'uuid': 'd7ea15b2-0664-1cd5-dc30-7c49a546ae80'}, ... ...
>>> role_names = map(lambda x: roles[x]["name_label"], roles)
>>> role_names.sort()
>>> print role_names

You can also find the same list of available roles by doing

cat /opt/xensource/debug/rbac_static.csv

in dom0.

[1] For instance, for installing the openldap client packages as the PAM/NSS provider for XCP in dom0, one can use yum:

# yum --disablerepo=citrix --enablerepo=base install openldap openldap-clients nss_ldap cyrus-sasl-ldap cyrus-sasl-gssapi

and then manually configure the PAM config file at /etc/pam.d/xapi, adding the relevant entries, and /etc/nsswitch.conf, adding the relevant ldap entries according to the nss_ldap and pam_ldap help entries in the manpages or in several detailed tutorials that can be easily found on the Internet, such as in and .