Tag Archives: Linux

Xen @ Build a Cloud Day Boston

As I mentioned in the Xen Day post, Xen.org was offered a slot at the Build an Open Source Cloud Day Boston. The Build a Cloud attendees were great. They were very engaged and asked lots of questions. The questions gave me a chance to cover several Xen topics that I tend to take for granted. In this post I’ll outline the talk that I gave and highlight some of the questions that were asked and discuss the answers.

The slides for the talk are available here: http://www.slideshare.net/deshantm/why-choose-xen-for-your-cloud

I decided to name the talk “Why Choose Xen For Your Cloud?” so that I could cover the history of Xen in the cloud, specific architectural advantages of Xen, and then cover XCP and Project Kronos. I don’t think that a lot of people realize that “Global Public Computing”  (or Cloud Computing as we know it today) was an idea that led to Xen. In other words, the Xen hypervisor was the solution to the need to break up, isolate, and provide accounting for system resources. Xen was born in the cloud. It’s no surprise then that Amazon EC2, Slicehost (now Rackspace Cloud Servers), and many others have chosen Xen as the basis of their clouds.

As I mentioned about Xen Dom0 support getting into the mainline Linux kernel, I got my first question (something like) “Isn’t Xen support in the Linux kernel going away due to KVM?”. To which I answered no, but let me clarify.  The question makes more sense for someone who is both a Linux expert and virtualization novice since the Linux kernel community is very strict about what it allows into the kernel. For instance, multiple implementations of the same functionality will not be accepted, but an abstracted version that can support more than one specific implementation is often required (OpenVZ is not in mainline, but Linux container components, often inspired by OpenVZ concepts, are included) . The main reason that Xen Dom0 support and KVM are both in the Linux kernel is that Xen and KVM have different architectures. Xen is a type-!, stand-alone hypervisor and KVM is a type-2, integrated hypervisor (I dig into these concepts again soon). Next to further clarify, Xen the hypervisor will never be included in Linux nor is it intended to be. When Xen was first created the Linux kernel at the time was used as a basis, but in a very special way (as explained by Keir Fraser) – the kernel was cut horizontally and the low level bits became the Xen hypervisor and the top part became the first Dom0. The Xen hypervisor is first to the hardware, followed by guest domains (including Dom0), by intentional design and is a fundamental requirement for being a type-1 hypervisor.

Next, as I covered the basic architectural design of a Xen system and its security benefits I was asked to clarify the difference between a type-1 and type-2 hypervisor. I find that the best way to explain this in a practical way that others can quickly understand is by using VMware ESX (the bare-metal, type-1 hypervisor from VMware) and VMware Workstation (the workstation class, install on top of an existing OS, type-2 hypervisor from VMware) since VMware is very well known and people can understand the difference. This question and answer are a perfect lead-in for the architectural and security advantages of Xen. In short, Xen has a small, well-defined, clean, and disaggregatable trusted computing base. This is in contrast to type-2 solutions, but also in contrast to type-1 hypervisors that put more services (perhaps than necessary or desired) into the hypervisor itself. Xen has unique security advantages primarily due to its well-architected design.

Since we were given an hour and fifteen minute slot, the XCP slides are fleshed out a bit more than I have presented in the past. Fortunately, our Xen Day team had prepared some really great materials and I was able to easily leverage those materials and re-focus them for the Build A Cloud/USENIX LISA audience. In particular, it is important to note the XAPI class diagram and cover the basic structure in some detail. The idea being that it helps system admins become familiar with the terminology they will need to master the xe command, but also that these devops-minded admins will likely be interested in writing scripts/tools that make direct XAPI calls. XAPI is great API for the cloud.

The rest of the XCP discussion is centered around why XCP is a great cloud platform. One of the key concepts that XCP, Open vSwitch, and the management tools address is mult-tenancy (multiple independent entities that need to share the same cloud and yet be isolated). XCP really shines for this cloud-minded audience and at least a few people were convinced to skip out on some of the Build a Cloud Day sessions and attend some of the Xen Day afternoon sessions to find out more about XCP. Xen Day took a deep dive into XCP.

XCP and cloud orchestration is an incredible story especially considering that XCP 1.0 was released in 2011. XCP is used in the first commercially available OpenStack cloud from Internap. CloudStack, which is a mature and feature-rich product itself, has also added support for XCP. OpenNebula also recently added support for XCP.  XCP is a great de-facto open source platform to build a cloud on.

I’ve spent a lot of time (and presented in more detail) presentations on Project Kronos, which in a very short time has been developed and is nearing a initial release. Stay tuned to the blog for updates on Project Kronos status. Current documentation on installing and using Project Kronos is available on the Xen wiki:


Xen Day Boston

It was a cloudy day in Boston last week as Xen.org hosted Xen Day Boston and CloudStack hosted Build an Open Source Cloud Day Boston.

Xen Day

First and foremost a big Thank You! to our Xen Day presenters: Patrick F. Wilbur, Steven Maresca, and Josh West. What a great and diversely talented team they were. I look forward to working with you all again soon.

You can read their bios on the Xen Day page, but let me tell you a bit about these guys and what makes them special. Pat lead off the day by presenting a condensed version of a Xen and XCP tutorial that is typically a day long tutorial itself.  Pat comes from an academic background with a lot of experience running training sessions and more famously as a co-author of “Running Xen”. His research is interesting because it takes a closer look at using virtual machines (virtual appliances more specifically) to provide a secure end-user system (secure academically, not just something to say in a commercial ;)).

Steve presented next about the Nuts and Bolts of XCP (especially its toolstack – XAPI). Steve has a day job as a systems analyst in an IT network security group at UConn. Perhaps more famously he is the lead developer of Zentific. Xen is his virtualization platform of choice because he can do *anything* with it. Steve did an amazing job covering the inner workings of XAPI, and why it is a great API. He also showed how easy it is to code against it  and provided example python scripts. He also described in detail how to use PXE to do automatic installs of XCP and guest VMs. The role-based access control (RBAC) explanation was also very good. (For more information see Marcus’ post on RBAC). Steve’s developer and systems background was a perfect match for the Xen Day as it was co-located with USENIX LISA ’11.

After an interactive demo session from the team –  installing XCP and doing a live demonstration of some of the XAPI possibilities, Josh West was up next describing a series of enterprise functional areas (Storage, High Availability/Fault Tolerance, Networking, and the Cloud) and their relationship to XCP. In other words, From the Data Center to the Cloud. Josh worked previously at TripAdvisor and set up a really neat XCP test lab for them. Josh’s real-life system administrator experience coupled with his passion for virtualization, the cloud, and Xen/XCP made his presentation of practical, real world use-cases amazing (some oohs and ahhs from the Xen Day attendees!).

Overall it was great to finally see some training material that is primarily focused on XCP!

The slides are available here:


We did not record the sessions, but we agreed that creating video tutorials and publishing these online later would be the better option. Watch the blog for more details!

Finally, we’d like to also say thank you to Citrix Bedford for providing hardware for use during Xen Day and also TripAdvisor for letting us use their test lab for preparation of the materials. (More on TripAdvisor and Xen Day). See also the video on the Xen Day page, where Ryan Pugatch goes into detail of how TripAdvisor is using XCP and why.

Build an Open Source Cloud Day

I’ll save the Build an Open Source Cloud details until another day, but until then - my slides here.

XCP, RBAC and PAM authentication in the XenAPI

Many users have been asking a bit about the internals of the authentication mechanisms in XCP and XAPI, and how to add extra users, so I thought I should talk a bit about it.

Just after installation, XCP allows only root to log in, since by default this is the only user available over PAM/NSS in dom0. Extra users can be added locally to /etc/passwd or externally by using NSS/PAM extensions such as LDAP [1], NIS, winbind, Likewise etc in dom0. However, due to backwards-compatible behavior in the XenAPI that goes back to XenServer 5.0, all these users will still have root access by default, and therefore they will all be allowed access to all of the XenAPI calls.

In order to restrict the XenAPI calls for these extra users, it is necessary to make XAPI aware that an external authentication mechanism such as PAM is being used instead of the default backwards-compatible internal XAPI authentication. This can be accomplished by the following xe cli command, which will reroute all the XenAPI authentication requests to the external PAM XAPI plugin:

# xe pool-enable-external-auth auth-type=PAM service-name=<any-name>

A nice side-effect of enabling external authentication in XAPI as above is that all the XenAPI authentication requests will go through XAPI’s role-based access control (RBAC) mechanism. The RBAC in XAPI allows fine-tuning the permissions of any extra users or groups that are available over external authentication.

In XCP parlance, external users and groups are called “subjects”. By default all subjects will be denied access to the XenAPI. In order to allow any of them access to a subset (or the full set) of the XenAPI calls, two steps are necessary:

1. Add a subject to the subject-list in the XAPI database:

# xe subject-add subject-name=<external-user-or-group-name>

By default, a new subject will not be allowed to use any XenAPI calls.

2. Assign a set of XenAPI calls to the subject:

# xe subject-role-add role-name=<role-or-permission-name>

The pre-defined role names are pool-admin, pool-operator, vm-power-admin, vm-admin, vm-operator, read-only, which define a total order of permission sets ranging from all XenAPI calls (pool-admin) to only read-only calls (read-only). A relatively up-to-date table relating the available roles to permissions can be found at http://support.citrix.com/article/CTX126441.

Roles are only alias to a set of XenAPI calls. It is also possible to assign more than one XenAPI call to a subject. Therefore, it is possible to tweak the pre-defined roles by adding extra calls to it. For instance, to create a vm-operator that can also migrate VMs (whose XenAPI call is by default only available to pool-operator), you can do the following, assuming you have a subject called “migrate-op”:

# xe subject-role-add role-name=vm-operator uuid=<uuid-of-subject-"migrate-op">
# xe subject-role-add role-name=vm.pool_migrate uuid=<uuid-of-subject-"migrate-op">

You can then check what roles are assigned to what subjects:

# xe subject-list

The user root is still available for disaster-recovery purposes, in case either the external authentication mechanism stops working due to some problem in the external authentication plugin or the pool master’s subject list is not available due to some accident. In this case, slave emergency commands to recover the host or the pool can still use the user root.

Also, an interesting detail is that even though the XenAPI calls are usually available through the master only, you can call the XenAPI session.login_with_password(username,pwd) on any slave so that you can distribute the authentication load through all the hosts of the pool.

It is possible to list the available roles using the xe cli command

# xe role-list

but this will result in a simplified list of the default role names. In order to obtain the full list of all the available role names that can be added to a subject, you need to use the XenAPI directly, for instance using python:

$ python
>>> import xmlrpclib
>>> x=xmlrpclib.Server("https://my-xenserver-host")
>>> s=x.session.login_with_password("root","my-password")['Value']
>>> roles = x.role.get_all_records(s)['Value']
>>> print roles
{'Status': 'Success', 'Value': {'OpaqueRef:d7ea15b2-0664-1cd5-dc30-7c49a546ae80': {'subroles': [], 'name_description': 'A basic permission', 'name_label': 'pool.get_wlb_enabled', 'uuid': 'd7ea15b2-0664-1cd5-dc30-7c49a546ae80'}, ... ...
>>> role_names = map(lambda x: roles[x]["name_label"], roles)
>>> role_names.sort()
>>> print role_names

You can also find the same list of available roles by doing

cat /opt/xensource/debug/rbac_static.csv

in dom0.

[1] For instance, for installing the openldap client packages as the PAM/NSS provider for XCP in dom0, one can use yum:

# yum --disablerepo=citrix --enablerepo=base install openldap openldap-clients nss_ldap cyrus-sasl-ldap cyrus-sasl-gssapi

and then manually configure the PAM config file at /etc/pam.d/xapi, adding the relevant pam_ldap.so entries, and /etc/nsswitch.conf, adding the relevant ldap entries according to the nss_ldap and pam_ldap help entries in the manpages or in several detailed tutorials that can be easily found on the Internet, such as in http://wiki.debian.org/LDAP/PAM and http://wiki.debian.org/LDAP/NSS .

XCP 1.1 has landed!

The XCP team would like to announce that we have released XCP 1.1. I wanted to thank community members who have tested RC1 for doing so and wanted to let you know that no significant issues have been found. Thus, we promoted the XCP 1.1 RC1 release candidate to final release status: so if you’ve already installed the RC1 release, there is no reason to upgrade.

New XCP Features
Here are some features included in XCP 1.1:

  • Security updates and bug fixes
  • IntelliCache: Enables you to use a combination of Shared storage and local storage caching.
  • Local Storage Spans All Physical Volumes: When EXT local storage is used on a host containing multiple physical disks, the local Storage Repository (SR) now spans all the disks in a single LVM volume group.
  • Reset-on-boot VM behaviour: Disks with the on-boot option set to reset, is now available for disks in any type of SR. (Previously it was only available for disks in NFS and EXT SRs.)
  • Enhanced guest OS support: Support for Red Hat Enterprise Linux (RHEL) 6.
  • For OpenStack and others: support for ebtables and other netfilter options have been added to the kernel. These options have been disabled by default, but can be re-enabled with simple sysctl commands. These are the same kernel changes that will be included the upcoming XCP 1.5 release and have been included in XenServer 6.0.
  • The Xapi version override feature has been added back in. This allows users to effectively rebrand their XCP boxes as XenServer hosts, in order to work better with products such as XenCenter.

Thank you!
Thanks again to all our beta testers. And extra thanks to everyone for being exceedingly patient with our long beta period! We’re in the middle of revamping our internal build system right now. Once we finish this work, XCP releases should be easier for us to produce. This work will also allow us to produce development snapshots of XCP releases in between stable point releases, to allow people to test new features as they make it into XCP’s and XenServer’s mainline.

I hope everyone enjoys using XCP 1.1. Keep an eye out here for news on the upcoming XCP 1.5 release schedule!

New Docs!
Note that new XCP documentation has been written on the wiki: thank you to the community members who have contributed. In particular I wanted to thank Andrew Eross from Locatrix for the new XCP HowTo documents. We still need to create a nice and easy to use index for XCP docs, but in the meantime you can easily find the docs by searching for XCP.

XCP 1.1 RC1 Release

The XCP team would like to announce that, after a long time in beta, we’ve released XCP 1.1 RC1, which we hope will become our final release. Here is a list of issues that we’ve resolved since the beta release:

  • The license expiry bug has been resolved. This was the cause of the DMC issues reported in the beta, so those issues are resolved as well.
  • For OpenStack and others, support for ebtables and other netfilter options have been added to the kernel. These options have been disabled by default, but can be re-enabled with simple sysctl commands. These are the same kernel changes that will be included in the upcoming XenServer 6.0 release.
  • The Xapi version override feature has been added back in. This allows users to effectively rebrand their XCP boxes as XenServer hosts, in order to work better with products such as XenCenter.
  • It is possible to upgrade from XCP 1.0 to XCP 1.1 RC1, but it is not possible to upgrade from the beta to the RC.

Thanks to all the people who participated in the beta program, and who filed bug reports for us. It’s always really exciting for us hear from people who are using XCP, and we hope that this release is the best version of XCP yet.

Introducing Thomas Goirand, XCP packager

I’d like to introduce you to Thomas Goirand, who has volunteered to help us package the XenAPI toolstack for Debian as part of the Kronos project. Thomas is a Debian package maintainer and CEO of GPLHost. Thomas attended DebConf in Bosnia two weeks ago, where he did a great deal of work for the XCP-to-Debian packaging effort. We asked Thomas to write a short summary of his history with Xen, and his Project Kronos packaging efforts to date.

Hi my name is Thomas Goirand and I am CEO of GPLHost and Debian maintainer.
GPLHost has been providing a VPS hosting service based on Xen, since Xen 2.07 and we have been extremely happy with Xen. In particular on performance, stability and feature richness. When the idea of cloud computing started to take off, GPLHost was thinking about providing a cloud computing service. But we were in the same situation as in 2004/2005, when there was only UML as hypervisor and there was no open source technology that was fitting the bill. So when OpenStack started as a project, it was quite natural that we wanted to contribute and offer Openstack to our customers.

OpenStack development was one of the rare instances, where Debian was downstream of Ubuntu as OpenStack is developed on Ubuntu first, closely following the Ubuntu release cycle. One would think that Opestack code for Debian and Ubuntu is identical: Not true! About 2-3 dozens of patches were necessary to have OpenStack Nova, Glance and Swift ready for a Debian upload. Examples of differences are: Ubuntu uses upstart, when Debian uses insserv. As a consequence, init scripts had to be written nearly from scratch. Anyway, this work is done and OpenStack is available in Debian SID, as well as in our private repository for the Squeeze backport. By the way, I am searching for co-maintainers for these important packages.

In its current state, OpenStack can only be used with KVM as hypervisor, if you want to use Debian. You can of course use OpenStack with XCP, which is a CentOS appliance. But since I am a Debian developer and all GPLHost infrastructure is based on Debian, running an RPM based distribution is not an option. So I investigated if it is possible to package Xen-API (xapi) for Debian. The answers I got from both the OpenStack and xen-devel list were quite disappointing: XCP was, from the beginning, designed as an appliance, with many core packages being modified (kernel, lvm, and other libraries…). Porting it would be hard, because a lot of packages would have to be modified. This would make it quite difficult to port without the help of upstream XCP (yes, I’m talking about Citrix here).

Later, someone told me that my questions on public mailing lists sparked some internal discussion inside the XCP project which is led by Citrix. This led to the creation of project Kronos, whose goal it is to make Xen-API independent of CentOS and to also move away from an appliance only model. The first goal is to have a full Xen-API support for Ubuntu and that “apt-get install xen-api” would magically install everything. But the work would be done in Debian first and synched from SID to Ubuntu (which is part of the normal release cycle for Ubuntu). I was delighted. Of course, as Ian Campbell told me at Debconf 11, “it was ready to burst”, and I believe it would have probably happen anyway, even without me asking. Nevertheless, I am happy I asked at the time and am also happy to work on the packaging.

Anyway, the XCP project started porting their code from the CentOS XCP appliance. Since then, I started the packaging work, together with the help of “upstream” XCP (as in: Mike McClurg, Jonathan Ludlam, and others at Citrix in Cambridge).

Currently, the following packages are more or less ready:

  • libblktap
  • Xen 4.1.1 (modified current packaging to build the Xen OCaml libraries)

The following will still need to be built:

  • blktap DKMS package (since blktap wont ever make it to kernel.org)
  • xen-api-libs
  • xen-api

Even though I don’t really know much about OCaml, I know enough about Debian packaging to help out. And thanks to the help of Ian Jackson and Ian Campbell, who are both Debian Developers working on Xen and thanks to the OCaml task force at Debian who all agreed to review the packages, I believe it will be possible to create packages with good quality straight from the start. For those of you who never wrote packages (being RPM or .debs), you got to understand that writing quality packages is a tedious task and that it really improves quality to have more than a pair of eyes looking at the result.

Since DebConf 2011 we have the Linux 3.0 kernel available in Debian, including upstream kernel.org support for Xen dom0 (others have already blogged about the inclusion of the dom0 features in kernel.org for the 3.0 release). Thanks you to the Debian kernel team and in particular to Ben Hutching who did this one upload. With a little bit of extra effort, it will soon be possible to run XCP directly from Debian, with Xen 4.1.1 and Linux 3.0, all of which are available right now in SID. This means that this support should be available in Ubuntu as well: it looks we may be too late for Ubuntu 11.10, so we are working towards having XCP support in Ubuntu for the 12.04 LTS release (well, in my case, I’m working for Wheezy support, not for Ubuntu…).

All this is very exiting, and I am really looking forward to the work being done, for the first tests, with XCP first and then with OpenStack.