Monthly Archives: February 2013

Xen is now officially in git

The Xen hypervisor is now officially in git.

There’s a single repo with a branch corresponding to each old xen*.hg tree:

  New git branch        Old mercurial tree
   master                xen-unstable.hg
   stable-4.0            xen-4.0-testing.hg
   stable-4.1            xen-4.1-testing.hg
   stable-4.2            xen-4.2-testing.hg
   staging               staging/xen-unstable.hg
   staging-4.0           staging/xen-4.0-testing.hg
   staging-4.1           staging/xen-4.1-testing.hg
   staging-4.2           staging/xen-4.2-testing.hg

Previously many (if not most) of the committers were using git themselves, and sometimes even inviting git-using contributors to provide their patch series as a public git branch.

While git’s user interface does have its critics (which include me!), its rich functionality, flexibility and performance are making it the power user’s dvcs of choice. Even if you accidentally shoot yourself in the foot (or vaporise your leg) due to some ill-considered command line syntax, there will be a way to recondense the vapour cloud back into a leg and stick it back on. As a friend of mine said recently, while you might either love or hate Marmite, with git many people both love and hate it.

So, anyway, after consultation and coordination, on Thursday we made the switch.

What used to be the official mercurial trees are now readonly mirrors. Contributors and downstreams who prefer to use mercurial can continue to use these, and of course submit patches using mercurial rather than git. So non-committers can indeed ignore this if they want to.

However, one benefit of officially switching to git is that there is now one very officially blessed git history. This means that it will be somewhat easier for git-using contributors to share patch series. But the main benefits are to committers.

The automatic testing system has needed some reeducation to cope, and is still suffering from some recidivism. The push gate for Xen 4.1 broke when it tried to push, and the bisector has been very confused. But this and any other problems should be sorted soon.

Personally, with my committer hat on, I’m already enjoying the convenience of having a single git tree containing all the Xen branches I deal with. And I’ve found that git’s tools for extracting patches from email and applying them are an improvement over what I was using before.
Continue reading

Improving event channel scalability

As machines are getting more and more powerful today, people want more from the powerful hardware. From a cloud user’s perspective, it is better to run more virtual machines on a single host. Xen currently supports running hundreds of guests on a single physical machine, but we plan to take it to the next magnitude – to run thousands of guests on a single host. To achieve this goal, we need to tackle several bottlenecks, ranging from the hypervisor to the guest kernel to the user space tool stacks. One of the most notable / tricky bit of all the improvements is certainly the event channel scalability, which should enable Dom0 / driver domains to handle more events than before, thus supporting more guests.

What exactly is event channel? In short, event channel is a primitive provided by Xen for event notifications. A typical guest like Linux can map four kinds of events into event channels: Inter-domain notification, IPI (Inter-processor interrupt), VIRQ (virtual IRQ) and PIRQ (physical IRQ).

Xen models devices into a frontend, which runs in the guests, and a backend, which runs in Dom0 or driver domains. The notifications between frontends and backends are done via inter-domain event channels, which means if we run out of event channels we cannot support any more guests. A typical guest equipped with one virtual network interface (or vif) and one virtual disk (or vbd) consumes four event channels in backend domain, that is one for console, one for xenstore, one for vif and one for vbd. We can come up with a rough estimation that a backend domain can support up to (NR_EVENT_CHANNELS_SUPPORTED / 4) guests, which doesn’t even account for the IPIs, VIRQs and PIRQs a backend domain may have. The NR_EVENT_CHANNELS_SUPPORTED for a 64 bit backend domain is 4096 whilst for a 32 bit backend domain is 1024, yielding less than 1024 guests supported by 64 bit backend domain and even less by 32 bit backend domain. So currently, we can support hundreds of guests at best. It gets worse when a guest has several vifs and vbds.

To improve the scalability of event channel, two designs are proposed, one is 3-level event channel which extends the current 2-level mechanism and the other is a ground-up FIFO-based approach. The 3-level design comes with a prototype (Xen patches, Linux patches) while the FIFO-based event channel ABI is currently under discussion on the mailing list (PDF version of draft V2), the final decision has not been made yet. But it is worth mentioning the undergoing work to the public.

The 3-level event channel design takes advantage of the existing 2-level design, simply extending event channel up to next level, which is straightforward. This design supports up to 256K event channels for a 64 bit backend domain and 32K event channels for a 32 bit backend domain, which in practice is more than enough in the long run. The downsides of this design is that:

  1. it uses global mapping address, which is only 1G
  2. a flat bitmap of events provides no priorities

Solving problem 1 is straightforward. DomU will never use so many event channels, so we just enable 3-level event channel for Dom0 and driver domains. Problem 2 is by design not easily solvable.

The FIFO-based design starts from scratch, which has following advantages over the 3-level design:

  1. event priorities
  2. use FIFO to ensure events fairness
  3. flexible / configurable event channel number for individual guest to reduce memory footprint
  4. consistent ABI across 32 / 64 bit guest / hypervisor

The main downside for FIFO-based design is, well, it is not yet implemented. Certainly many details need to be pinned down (such as avoid using too much global mapping address) and benchmarks need to be conducted. Event handling code is complex by natural and we should be patient to wait for this design to become mature.

From a user’s point of view, either of the two designs should improve event channel scalablity and take Xen’s ability to support guests to next magnitude. 3-level design is proposed to meet the time frame of 4.3 release because of its simplicity. The FIFO-based design is proposed since we are defining a new ABI and our developers think it is high time we start from scratch. The door is still open, feel free to share your views, concerns and suggestions on Xen development list.

Upcoming Events: Build a Cloud Day, SCaLE 11x, Document Day, Linaro Connect Asia

Build a Cloud Day, February 22, Los Angeles, CA, USA

As last year, a Build a Cloud Day will be held at SCaLE 11x, where Joe Brockmeier will give an overview of Xen and XCP.

SCaLE 11x, February 23 – 24, Los Angeles, CA, USA will also again be at SCaLE 11x this year with a Xen and XCP Update and a booth. Russell Pavlicek, our new Evangelist and I will be at the booth. Drop by and say hello!

Xen Document Day: February 25

Another Xen document day is coming up next Monday. Xen Document Days are for people who care about Xen Documentation and want to improve it. Everybody who can and wants contribute is welcome to join! All you need is to join the Xen wiki and hang out on the #xendocs IRC channel.

For a list of items that need work, check out the community maintained TODO and wishlist. We have a few beginners items in the list. And everybody is welcome to add to the TODO list. Of course, you can work on anything you like: the list just provides suggestions.

See you on IRC : #xendocs @ freenode ! And have fun!

Linaro Connect Asia : March 4 – 8, Hong Kong will have a presence at LCA 13. Quite a few members of the Xen community will be there: Stefano Stabellini, Ian Campbell and me. There will be a number of talks:

Besides the Xen specific talks, there are many more sessions relevant to Xen. Looking forward to see you in Hong Kong.

An Open Source Prodigal Returns

It’s not often in life that you get a chance to do what you love — twice.

A few years ago, I was given the opportunity to spend my days working with and talking about Open Source software.  It was exhilarating while it lasted, but after a few years, I had to return to product-based professional services to make a living.

Fast forward another nine years and the opportunity to work with and talk about Open Source has returned — and I couldn’t be happier about it.

Pengi, courtesy of

For those of you who didn’t run into me the first time around, let me introduce myself.  My name is Russell Pavlicek.  I am a passionate believer in Open Source software.  And I have a really big mouth.

In 1995, I was working in Professional Services for the now-defunct Digital Equipment Corporation (eaten by Compaq, which was, in turn, eaten by HP) when I became involved with an effort requiring quick development of Unix skills.  I wasn’t much of a Unix guy, so when I saw an ad in a computer catalog for a PC-based Unix called “Linux” (Yggdrasil Plug-n-Play, Nov 1994 edition to be specific), I jumped at the chance to order it.  A day later, when the team involved in the aforementioned effort covened, I discovered that most of my coworkers needed additional Unix skills as well. In a moment blessed with tinges of both irony and prophecy, I shared my discovery of Linux with them. The ironic element was that I was advocating software I had never used and knew nothing about.  The prophetic element was that this advocacy of Open Source would characterize my professional direction for the years which followed.

Continue reading

Xen 4.3 mid-release roadmap update

At XenSummit in August, I talked about the new planning process that is experimenting with. This is apparently a pretty hot topic, as that presentation on has received over 72,000 views! We’re about half-way through the planned 4.3 development cycle, and while I’ve been regularly tracking the status of various features on the xen-devel mailing list, and also updating them on the 4.3 roadmap wiki page, I thought it might be useful just to give an update on how things are progressing.

Continue reading

FOSDEM’13 : Personal Impressions

This year’s FOSDEM is now over, and I finally made it back home. FOSDEM was buzzing, there was great content, lots of beer, meeting old friends and makign new friends. The number of attendees was impressive: according to the organizers – all volunteers – more than 5,000 open source enthusiast attended. Probably the number was even higher as FOSDEM requires no registration.

This year, I co-organized the virtualisation devroom with David Neary and Renzo Davioli. Thank you to both of them, the speakers, the audience and the FOSDEM organizers who helped made the devroom a success – at least in my view. As I am thanking people, I also wanted to thank our volunteers from the Xen developer community and user community who submitted talks and spent time on our booth. A special thank you goes to KB from CentOS, who saved our day by helping us out with CentOS-Xen co-branded T-Shirts. Our own T-Shirts didn’t arrive.

Continue reading