Monthly Archives: May 2014

June 2014 Brings Bumper Crop of Xen Project Talks to the US

The month of June has a variety of Xen Project talks available to conference attendees across the United States:

CentOS Dojo, Cincinnati Ohio, June 4

First, for people in the Cincinnati Ohio area, there is the CentOS Dojo meeting on Wednesday, June 4.  At 1:30 PM, Xen Project Evangelist Russell Pavlicek will speak on “Using and Understanding Xen4CentOS.”  Registration for the event can be found on the CentOS Dojo site.

Texas Linux Fest, Austin Texas, June 13-14

For those who can’t make it to Ohio, Russell will be reprising the Xen4CentOS talk at Texas Linux Fest conference during the Lightning Talks.  Be there on Saturday evening, June 14 to take it in.  And while you’re there, drop in on Russell’s Open Source talk “Geek Empowerment: The Real Heart of Open Source” at 1:30 PM.  Registration costs as little as US$30 for the entire event, so don’t miss out if you are in the greater Austin area.

SouthEast LinuxFest, Charlotte North Carolina, June 20-22

And finally, folks near Charlotte, NC can attend two different Xen Project talks at SouthEast LinuxFest.  On Friday June 20 at 9 AM, listen to Russell as he presents “Introduction to the Security Features of the Xen Project Hypervisor.”  Then stick around until Sunday at 1:30 PM to learn about “Xen Project 4.4 Features and Futures.”  Plus, Russ will be delivering his “Geek Empowerment” talk on Sunday at 10:15 AM.    The SELF conference is community run and free of charge, so register today!

And don’t forget to submit talks to the Events Calendar

Do you know of other Xen Project presentations coming up in the weeks ahead?  They won’t appear in our events calendar unless you post them there.  Login to and submit the information to let others know about more opportunities to learn about Xen Project.

Welcoming Rackspace as Xen Project Advisory Board member

We’re excited to welcome Rackspace as our newest Xen Project Advisory Board member today. Since becoming a Linux Foundation Collaborative Project about a year ago, we’ve announced four new members, including ARM, NetApp and Verizon Terremark. Rackspace

Rackspace has used Xen Project virtualization software since 2006. The company’s public cloud business supports hundreds of thousands of Xen virtual machines and millions of snapshots with the software. Global deployments aren’t slowing down anytime soon, according to Paul Voccio, Senior Director of Software Development at Rackspace. New machines ship in weekly, while new clusters come online every few days, according to Voccio.

“Since we’ve been working with Xen on a much deeper level than in the past, we thought the time was right to contribute back to the project,” he said. “Becoming a member is the first step in that process, which will allow us to become more involved with the project.”

Speaking to us from last week’s OpenStack Summit Atlanta, Voccio and his colleague Antony Messerli, a Principal Engineer, are already aggressively promoting Xen as a type 1 hypervisor within the open cloud community his company helped create.

Because Rackspace runs OpenStack and XenServer, which contain both the Xen Project Hypervisor and Xen Project XAPI, the two technologies are battle proven to work together in large-scale environments right out of the box, said Messerli. OpenStack fully supports XAPI integration with the Xen hypervisor as well as open source XenServer.

Rackspace has also recently worked with the CentOS community to get the Xen Project hypervisor tested and stabilized on CentOS.

“Currently we’re working with Citrix and the open source community to get XAPI tested and working on distributions like Debian and CentOS so that it’s easier for users that need to run their own distribution to consume Xen and OpenStack,” said Messerli.

Messerli said he believes Xen Project technology remains relevant because it has proven its reliability and stability through the years running some of the largest public clouds and varying workloads. Its high performance and security for multi-tenancy are critical for cloud computing. Moving forward, Voccio identified some near-term priorities where he hopes his team can add value.

“We’ll be looking to make performance improvements related to networking and disk I/O, as well as improving support for more distributions,” Voccio said. “We also plan to work with upstream Xen projects to establish a better testing path.”

Later this month Rackspace’s UK office will host a sold-out Xen Project Hackathon, May 29 and May 30 in London. Always an open and welcoming “host,” Rackspace has supported the Xen Project community by hosting our web site, wiki, mailing lists, blog and other services since 2011. The following links provide more information on Rackspace’s use of Xen Project technology:

HowTo: Xen4CentOS

Using CentOS 6 as the Control Domain for the Xen Project Hypervisor

Lots of hosting companies and other service providers settled on combination of the Xen Project Hypervisor and CentOS as a platform for their virtualization needs.  Unfortunately, CentOS 6 brings with it the same kernel as RHEL 6 — a kernel which lacks support for the Xen Project Hypervisor.

Established users had a problem: either give up their hypervisor, or give up the distribution which manages it.  But one of  the beauties of Open Source software is that modification of the software stack is always an option.  And this brought about the birth of a new effort to restore the hypervisor to the distribution via a project called Xen4CentOS.


Continue reading

Announcing the Program Management Committee for the Xen Project Developer Summit

The Xen Project Developer Summit is approaching: the Call For Participation will be open for two more days until May 16, 2014 11:55pm (EST).

Our Program Management Committee

I wanted to also take the opportunity to introduce this year’s Program Management Committee.

  • Amir Chaudhry (University of Cambridge): Amir is a post-doc at the Cambridge Computer Lab. Amir is program manager at OCaml Labs and runs community outearch activities in Mirage OS, a Xen Project team.
  • Boris Ostrovski (Oracle): Boris is working on various Linux and Xen Project components and is also maintainer of a number of Xen project subsystems. He is also a Google Summer of Code Mentor.
  • Dario Faggioli (Citrix): Dario has interacted with the Linux kernel as part of his PhD working on real-time scheduling and other embedded technologies. He now works on various Xen Project components and is the Xen Project Blog Czar.
  • Lars Kurth (Chairman of the Xen Project Advisory Board): Lars has been working as Community Manager for the Xen Project for 3 years now and also chairs the Xen Project Advisory Board and other Xen Project Working Groups.

Developer Summit Program Announcement

We are aiming to publish the Xen Project Developer Summit program in the 1st week of June. People who have submitted talks, should get an acceptance e-mail a week before.

Birds of a Feather Sessions & Discussion Groups

This year we will again have space for Birds of a Feather Sessions & Discussion Groups. We will publish how you can request a BoF a little bit closer to the event. In the meantime you should be aware of the ground rules for BoFs:

  • Each BoF host will get 3-5 minutes (depending on the number of BoFs on the day) to pitch your BoF to the entire audience. Slides are not allowed.
  • After we publish the Xen Project Developer schedule, community members that have registered for the summit can submit a request to host a BoF (specifying a couple of slots in preference order)
  • BoFs are small discussion groups, not presentations. You are expected to take notes (or nominate an attendee to do so) and post discussion notes on one of our mailing lists after the summit.

Developer Meeting

I am also pleased to announce that we will also be hosting a 1/2 day Xen Project Developer Meeting the day after the Xen Project Developer Summit. Spaces are limited: the event is open to all members of the Developer Community. More details will follow soon.

Where to stay at the summit

Discounted hotels are listed at the event website at the price of 199 USD per night including wifi. Reservations have to be made by July 30th. We are sharing a room block with other Linux Foundation events, so please book early.

How the Hypervisor team manages releases

With the Xen Project Hypervisor 4.4 released a bit more than a month ago, the project is planning for version 4.5, so it’s a good time to outline how we manage releases.

New Release Manager: Welcoming Oracle’s Konrad Rzeszutek Wilk

But first I want to thank George Dunlap for successfully managing the 4.3 and 4.4 releases of the Xen Project Hypervisor. The Release Manager role is a volunteer role open to maintainers. George has stepped down from his role recently to find more time for coding and help bootstrap the CentOS virtualization SIG.

Konrad Rzeszutek Wilk has volunteered to take on the role for the 4.5 release. A big welcome and Thank You!

Konrad is Software Development Manager at Oracle. His group’s mission is to make Linux and Xen Project virtualization better and faster. As part of this work, Konrad has been the maintainer of the Xen Project subsystem in Linux, Xen Project maintainer and now also Release Manager for the 4.5 release of the Xen Project Hypervisor. Konrad has been active in the Linux and Xen Project communities for more than 6 years and was instrumental in adding Xen Project support to the Linux Kernel.

How we manage releases

As is the case for many open source projects, the Xen Project community does not maintain a committed roadmap as proprietary software vendors do. However, we do strive to accurately track development for new releases, with a predictable release cadence for major releases and maintenance releases. We aim to release the Xen Project Hypervisor every 6-7 months; historically we had a release cycle that ranged from 9 to 18 months.  The new Release Manager role has already proven instrumental delivering faster turnaround and predictability.

How contributors operate

Our best practice recommends that new features that intend to go into the next release are initially announced and discussed on the xen-devel mailing list. This eliminates the risk of multiple developers starting to implement a feature at the same time. The Release Manager will usually follow up and ask whether the feature should be included into the backlog.

A monthly community call is a forum for raising issues and discussing  bigger development items. We also run a Hackathon once a year (typically in spring) and a 1/2 day, face-2-face meeting with all the core developers before or after our Developer Summit. These in-person meetings are usually used to bootstrap the planning process for a release.

How we create a Roadmap

During planning and development of a release, the Release Manager sends out monthly development update e-mails on xen-devel that are labelled Xen x.y Development Update (check out the 4.4 mails). In these mails, we ask developers:

  • whether they have items to be tracked on the roadmap and if so to add them to a backlog;
  • for status updates, when they are on the backlog; and
  • how likely it is that a feature will be in our codeline (including reviews) by the next update.

Toward the end of the development cycle, we start to treat blocker and critical bugs in the same way. More public announcements on release status are made on our blog and the announce list at critical points in the release cycle.

Stages within a Release Cycle

The following diagram shows the key stages, gates and git branches in our release cycle.


Feature Development / Feature Freeze: Feature development starts as soon as the release branch for the previous release has been created. It ends at the Feature Freeze point, when we typically won’t allow new features into the master branch. Patches that were submitted before Feature Freeze and are being reviewed may still be let in based on a risk analysis. This phase is typically 3-4 months long. The timing is adapted based on major holidays (XMas and New Years) or major conferences, such as Xen Project Developer Summits.

Hardening / Code Freeze: During the hardening phase, which typically lasts 3-4 weeks, we complete any remaining reviews of patches applied during development and focus on bug fixes. This phase ends with the Code Freeze point, after which each bug will be considered on a case-by-case basis and risk to the quality of the release. Major vendors will start running their complete test suites during this phase of development. Hardening ends with the first release candidate (RC1).

Release Candidates / Release: We close the release cycle making a number of release candidates (RCs). Historically, we needed 4-6 RCs and create one every 1-2 weeks. During the Release Candidate phase, we run Xen Project Test Days and major vendors in the community run their test suites against RCs. Typically, only bugs of blocker severity are fixed. This phase typically lasts 6 weeks, but may be extended if blocker bugs are found that need to be addressed. We end this phase by creating a stable release branch.

Release Announcement: Usually, we make an official Release Announcement shortly after the stable release branch has been created. But sometimes, we delay by a few days to coordinate press activities.

Improving the Process

We generally have discussions on how to improve the release process, at face-2-face meetings such as the Hackathons and Developer meetings. At previous developer meetings, we decided to focus on improving our test infrastructure and started using Coverity Scan regularly on our codebase and formed the test framework working group. In the upcoming May Hackathon, we will for example have a discussion about how to handle the ever-increasing traffic on our development list (which regularly hits more than 4,000 messages per month).

Additional Information

Summary of the gains of Xen oxenstored over cxenstored

This week, we are reblogging this excellent piece from Luis from SUSE. The article came about because of a discussion Luis had at the Linux Foundation Collaboration Summit in Napa, and he decided to write down some basic generals of the xenstore, a review of its first implementation and a summary of oxenstored. The original post is available here, on Luis’ blog.

OXenstored is the default in Xen, only if the ocaml compiler was installed at build time. This means that in many Linux distributions that ship Xen, cxenstored is most likely your default. You can check whether oxenstored is installed by checking whether the oxenstored process runs in your Dom 0.

It all started with a paper: OXenstored – An Efficient Hierarchical and Transactional Database using Functional Programming with Reference Cell Comparisons

First a general description of the xenstore and its first implementation. The xenstore is where Xen stores the information over its systems. It covers dom0 and guests and it uses a filesystem type of layout kind of how we keep a layout of a system on the Linux kernel in sysfs. The original xenstored, which the paper refers to a Cxenstored was written in C. Since all information needs to be stored in a filesystem layout any library or tool that supports designing a tree to have key value store of information should suffice to upkeep the xenstore. The Xen folks decided to use the Trivial Database, tdb, which as it turns out was designed and implemented by the Samba folks for its own database. Xen then has a daemon sitting in the background which listens to reads / write requests onto this database, that’s what you see running in the background if you ‘ps -ef | grep xen’ on dom0. dom0 is the first host, the rest are guests. dom0 uses Unix domain sockets to talk to the xenstore while guests talk to it using the kernel through the xenbus. The code for opening up a connection onto the c version of the xenstore is in tools/xenstore/xs.c and the the call is xs_open(). The first attempt by code will be to open the Unix domain socket with get_handle(xs_daemon_socket()) and if that fails it will try get_handle(xs_domain_dev()), the later will vary depending on your Operating System and you can override first by setting the environment variable XENSTORED_PATH. On Linux this is at /proc/xen/xenbus. All the xenstore is doing is brokering access to the database. The xenstore represents all data known to Xen, we build it upon bootup and can throw it out the window when shutting down, which is why we should just use a tmpfs for it (Debian does, OpenSUSE should be changed to it). The actual database for the C implementation is by default stored under the directory /var/lib/xenstored, the file that has the database there is called tdb. On OpenSUSE that’s /var/lib/xenstored/tdb, on Debian (as of xen-utils-4.3) that’s /run/xenstored/tdb. The C version of the xenstore therefore puts out a database file that can actually be used with tdb-tools (actual package name for Debian and SUSE). xentored does not use libtdb which is GPLv3+, Xen in-takes the tdb implementation which is licensed under the LGPL and carries a copy under tools/xenstore/tdb.c. Although you shouldn’t be using tdb-tools to poke at the database you can still read from it using these tools, you can read the entire database as follows:

 tdbtool /run/xenstored/tdb dump

The biggest issue with the C version implementation and relying on tdb is that you can live lock it if you have a guest or any entity doing short quick accesses onto the xenstore. We need Xen to scale though and the research and development behind oxenstored was an effort to help with that. What follows next is my brain dump of the paper. I don’t get into the details of the implementation because as can be expected I don’t want to read OCaml code. Keep in mind that if I look for a replacement I’m looking also for something that Samba folks might want to consider.

OXenstored has the following observed gains:

  • 1/5th the size in terms of line of code in comparison to the C xenstored
  • better performance increasing support for the number of guests, it supports 3 times number of guests for an upper limit of 160 guests

The performance gains come from two things:

  • how it deals with transactions through an immutable prefix tree. Each transaction is associated with a triplet (T1, T2, p) where T1 is the root of the database just before a transaction, T2 is the local copy of the database with all updates made by the transaction made up to that point, p is the path to the furthest node from the root T2 whose subtree contains all the updates made by the transaction up that point.
  • how it deals with sharing immutable subtrees and uses ‘reference cell equality’, a limited form of pointer equality, which compares the location of values instead of the values themselves. Two values are shared if they share the same location. Functional programming languages enforce that multiple copies of immutable structures share the same location in memory. oxenstored takes avantage of this functional programming feature to design trie library which enforces sharing of  subtrees as much as possible. This lets them simpilfy how to determine and merge / coalesce concurrent transactions.

The complexity of the algorithms used by oxenstored is confined only to the length of the path, which is rarely over 10. This gives predictable performance regardless of the number of guests present.