Author Archives: Lars Kurth

About Lars Kurth

Lars Kurth is a highly effective, passionate community manager with strong experience of working with open source communities (Symbian, Symbian DevCo, Eclipse, GNU) and currently is community manager for Lars has 9 years of experience building and leading engineering teams and a track record of executing several change programs impacting 1000 users. Lars has 16 years of industry experience in the tools and mobile sector working at ARM, Symbian Ltd, Symbian Foundation and Nokia. Lars has strong analytical, communication, influencing and presentation skills, good knowledge of marketing and product management and extensive background in C/C , Java and software development practices which he learned working as community manager, product manager, chief architect, engineering manager and software developer. If you want to know more, check out Personally, Lars has a wide range of interests such as literature, theatre, cinema, cooking and gardening. He is particularly fascinated by orchids and carnivorous plants and has built a rather large collection of plants from all over the world. His love for plants extends into a passion for travel, in particular to see plants grow in their native habitats.

Introducing the Xen Project Code Review Dashboard

The Xen Project’s code contributions have grown more than 10% each year. Although growth is extremely healthy to the project as a whole, it has its growing pains. For the Xen Project, it led to issues with its code review process: maintainers believed that their review workload increased and a number of vendors claimed that it took significantly longer for contributions to be upstreamed, compared to the past.

The project developed some basic scripts that correlated development list traffic with git commits, which showed indeed that it took longer for patches to be committed. In order to identify possible root causes, the project initially ran a number of surveys to identify possible causes for the slow down. Unfortunately, many of the observations made by community members contradicted each other, and were thus not actionable. To solve this problem, the Xen Project worked with Bitergia, a company that focuses on analyzing community software development processes, to better understand and address the issues at hand. We worked with Bitergia on an initial statistical analysis of the code review process and later on a Code Review Dashboard for use by the community. The following OSCON presentation lays out the journey, the project went through:

Findings of the Initial Code Review Study

There were three key areas that we found that were causing the slow down:

  • Huge growth in comment activity from 2013 to 2015
  • We also saw that the time it took to merge patches (=time to merge) increased significantly from 2012 to the first half of 2014. However, from the second half of 2014 time to merge moved back to its long term average. This was a strong indicator that the pre-emptive measures we took, such as a focus on design and architecture reviews and contributor training, actually had an effect.
  • Looking at this in more detail, it turned out that complex patches were taking significantly longer to merge than small patches. As it turns out, a significant number of new features were actually rather complex. At the same time, the demands on the project to deliver better quality and security had also raised the bar for what could be accepted, which impacted new contributors more than established ones.

Introducing the Xen Project Code Review Dashboard

To make the tooling that we developed more accessible to the entire community, the Xen Project Advisory Board funded the development of a Code Review Dashboard. We defined a set of use-cases and supporting data that broadly covered three areas:

  • Community use cases to encourage desired behaviour: this would be metrics such as real review contributions (not justed ACKed-by and Reviewed-by flags), comparing review activity against contributions.
  • Performance use cases that would allow us to spot issues early: these would allow us to filter time related metrics by a number of different criteria such as complexity of a patch series.
  • Backlog use cases to optimize process and focus. The intention here was to give contributors and maintainers tools to see what reviews are active, nearly complete, complete or stale.

To find out more check out

Value for other projects

Like many FOSS projects, the Xen Project code review process uses a mailing list-based review process, and this could be a good blueprint for projects that are finding themselves in the same predicament. It is already clear, that there are many useful additions that can in future be added to the technology we have developed. In addition, we are currently working on improvements of the Code Review Dashboard as part of an Outreachy Project by Priya V (check out her blog), which is jointly mentored by Lars Kurth (Xen Project) and Jesus M. Gonzalez-Barahona (Bitergia). All the code is open source: if you want to have a look, check out the code and the contribution guide.

Announcing Xen Project 4.7 RC and Test Day Schedule

Yesterday we created Xen 4.7 RC2 and will release a new release candidate every Wednesday, until we declare a release candidate as the final candidate and cut the Xen 4.7 release. We will also hold a Test Day every Friday for the release candidate that was released the Wednesday prior to the Test Day. This means we will have Test Days on May 13th, 20th, 27th and June 3rd. Your testing is still valuable on other days, so please feel free to send Test Reports as outlined below at any time.

Getting, Building and Installing a Release Candidate

Release candidates are available from our git repository at

git:// (tag 4.7.0-<rc>)

where <rc> is rc1, rc2, rc3, etc. and as tarball from<rc>/xen-4.7.0-<rc>.tar.gz<rc>/xen-4.7.0-<rc>.tar.gz.sig

Detailed build and Install instructions can be found on the Test Day Wiki.

Testing new Features, Test and Bug Reports

You can find Test Instructions for new features on our Test Day Wiki and instructions for general tests on Testing Xen. The following pages provide information on how to report successful tests and how to report bugs and issues.

Happy Testing!

Please Welcome new Members of the Xen Project Hypervisor Leadership Team

Evolution of Hypervisor Git Commits within the project. Note that in in the same time period, the number of individuals and organisations contributing to the project has nearly doubled.

Evolution of Hypervisor Git Commits within the project. Note that in parallel the number of individuals and organisations contributing to the project has nearly doubled.

The Xen Project has experienced incredible growth in our community (see diagram on the right) and simultaneously the Xen Project advisory board has funded a lot of great projects that help support the larger Xen Project ecosystem, for example MirageOS, a library operating system that constructs unikernels for secure, high-performance network applications across a variety of cloud computing and mobile platforms. These projects are extremely important to the expansion and betterment of virtualization and cloud computing infrastructure, but also demand more work to be done by committers and maintainers.

We understood that there was plenty of leadership among the community, but didn’t know the best way to promote contributors to maintainers and committer roles for leadership to the Hypervisor and its interface with Linux.

We decided to introduced a new convention, by which we actively reminded community members to nominate or self-nominate themselves for leadership roles. Often times, active developers simply worked on the project, but did not consider to nominate newcomers (or themselves) for these leadership roles within the project.

We also felt that running the entire nomination process in public, which may include public feedback on a nominee, could discourage people from recommending themselves or recommending others. So we decided to follow the approach that Debian uses for Technical Committee seats, where candidates are nominated in private. A group of the senior leadership for the Xen Project would then review the submissions and provide feedback and acceptance, making the overall process less intimidating.

As for committers and maintainers, we promoted several key Xen Project contributors to these distinguished roles based on previous experience and similar work they had already been performing, but didn’t have the authorship to own. This very important as it will ensure the group of maintainers that we currently have will have the support they need to accommodate for the Xen Project hypervisor.

Through taking the approach of consistent promotion and private forums, we have found some incredible new members to the Xen Project Hypervisor Leadership team that we want to introduce to you.

The following people have been elected to be new Committers to the project, they will be joining long-time committers Ian Campbell, Ian Jackson, Jan Beulich and Konrad Rzeszutek Wilk:

Andrew Cooper has been working on the Hypervisor since 2011 and has added a number of major new features such as Migration v2, significant change to trap handling, improvements to cpuid handling for guests and many more.

George Dunlap has been working on the Hypervisor since 2005 and was heavily involved in making the tracing system useable for performance analysis, optimising the shadow code, wrote the credit2 scheduler and developed many other significant features and improvements in the hypervisor. In addition, he was our first Release Manager and is leading the CentOS Virtualisation SIG within CentOS.

Stefano Stabellini has been working on the Hypervisor and the Linux Kernel since 2007 and was instrumental in bringing ARM support to the Xen Hypervisor. He has also been leading many other activities within the project, such as the creation of libxenlight, adding support upstream QEMU to Xen, Xen OpenStack integration and Raisin.

Wei Liu started to work on the Xen Project as a GSoC student in 2011 (working in virtio support). He has been working on libxl support, event channel scalability, MiniOS and many other major Xen features. In addition, he has been the Xen Project Release Manager since Xen Project 4.6 release.

Andrew and Wei celebrated their appointment at the Xen Project hackathon last month by submitting and ACKing a piece of code while on a punt on the river Cam in Cambridge, UK.

Security Team
In addition, Andrew Cooper and George Dunlap are now also members of the Xen Project Security Team, alongside Ian Jackson, Jan Beulich, Konrad Rzeszutek Wilk and Tim Deegan.

The following people were also recently added as MAINTAINERS of the project: Doug Goldstein (KConfig, Travis CI), Julien Grall (ARM support, device tree, …), Meng Xu (RTDS Scheduler) and Paul Durrant (x86 I/O emulation, x86 viridian enlightenments, …). In addition, we clarified some ambiguities around the maintainer role.

Linux Kernel Maintainers
Jürgen Gross who has been a Linux kernel and Xen developer since 2004, but has significantly increased his engagement within the community in the last two years, is now Linux Kernel maintainer for the Xen Hypervisor Interface alongside Boris Ostrovsky and David Vrabel. Other maintainers of Xen specific components in the Linux Kernel are Stefano Stabellini, Wei Lui, and Konrad Rzeszutek Wilk.

A couple of months ago two of our committers, Keir Fraser and Tim Deegan, formally stepped down in their roles as committers from the Hypervisor team. We want to thank Keir and Tim for the vast contributions to the project. We look forward to seeing what they work on next and, again, thank them for the success that they brought to the open source Xen hypervisor.

Xen Project Hackathon 16 : Event Report

We just wrapped another successful Xen Project Hackathon, which is an annual event, hosted by Xen Project member companies, typically at their corporate offices. This year’s event was hosted by ARM at their Cambridge HQ. 42 delegates descended on Cambridge from Aporeto, ARM, Assured Information Security, Automotive Electrical Systems, BAE Systems, Bromium, Citrix, GlobalLogic, OnApp, Onets, Oracle, StarLab, SUSE and Vates to attend. A big thank you (!) to ARM and in particular to Thomas Molgaard for organising the event and the social activities afterwards.

Here are a few images that helped capture the event:

Taking a breather and photo opp outside of ARM headquarters in Cambridge

Taking a breather and photo opp outside of ARM headquarters in Cambridge

Working on solving the mysteries of the world.

Working on solving the mysteries of the world

Continuing to work hard on solving the mysteries of the world

Continuing to work hard on solving the mysteries of the world

Xen Project Hackathons have evolved in format into a series of structured problem solving sessions that scale up to 50 people. We combine this with a more traditional hackathon approach where programmers (and others involved in software development) collaborate intensively on software projects.

This year’s event was particularly productive because all our core developers and the project’s leadership were present. We focused on a lot of topics, but two of our main themes this year evolved around security and community development. We’ll cover these topics in more detail and how they fit within our next release 4.7 and development going forward, but below is a little taste of some of the other themes of this year’s Hackathon sessions:

  • Security improvements: A trimmed down QEMU to reduce attack surface, de-privileging QEMU and the x86 emulator to reduce the impact of security vulnerabilities in those components, XSplice, KConfig support which allows to remove parts of Xen at compile time, run-time disablement of Xen features to reduce the attack surface, vulnerabilities, disaggregation and enabling XSM (Xen’s equivalent of the Linux Security Modules which are also known as SELinux) by default.
  • Security features: We had two sessions on the future of XSplice (first version to be released in Xen 4.7), which allows users of Xen to apply security fixes on a running Xen instance (aka no need to reboot).
  • Robustness: A session on restartable Dom0 and driver domains, which again will significantly reduce the overhead of applying security patches.
  • Community and code review: A couple of sessions on optimising our working practices: most notably some clarifications to the maintainer role and how we can make code reviews more efficient.
  • Virtualization Modes: The next stage of PVH, which combines the best of HVM and PV. We also had discussions around some functionality that is currently developed in Linux on which PVH has dependencies.
  • Making Development more Scalable: A number of sessions to improve the toolstack and libxl. We covered topics such as making storage support pluggable via a plug-in architecture, making it easier to develop new PV drivers to support automotive and embedded vendors, and improvements to our build system, testing, stub domains and xenstored.
  • ARM support: There were a number of planning sessions for Xen ARM support. We covered the future roadmap, how to implement PCI passthrough, and how we can improve testing for the increasing range of ARM HW with support for virtualization, also applicable outside the server space.

There were many more sessions covering performance, scalability and other topics. The session’s host(s) post meeting notes on xen-devel@ (search for Hackathon in the subject line), if you want to explore any topic in more detail. To make it easier for people who do not follow our development lists, we also posted links to Hackathon related xen-devel@ discussions on our wiki.

Besides providing an opportunity to meet face-to-face, build bridges and solve problems, we always make sure that we have social events. After all Hackathons should be fun and bring people together. This year we had a dinner in Cambridge and of course the obligatory punting trip, which is part of every Cambridge trip.

Embarking on the punting journey

Embarking on the punting journey

Continued exploration of discovering the mysteries of the universe, while on a boat

Continued exploration of discovering the mysteries of the universe, while on a boat

Again, a big thanks to ARM for hosting the event! Also, a reminder that we’ll be hosting our Xen Project Developer Summit next August in Toronto, Canada. This event will happen directly after LinuxCon North America and is a great opportunity to learn more about Xen Project development and what’s happening within the Linux Foundation ecosystem at large. CFPs are still open until May 6th!

Call For Participation for the Xen Project Developer Summit in Toronto


Now Accepting Submissions Through May 6

We’re excited to announce the call for speaking proposals for Xen Project Developer Summit 2016, which will be held in Toronto, Canada, August 25-26, 2016. The Xen Project Developer Summit brings together the Xen Project’s community of developers and power users for their annual developer conference. The summit will be co-located with a number of other events, including LinuxCon, ContainerCon, KVM Forum and Linux Security Summit.

To get a sense of past accepted submissions, check out last years presentations. Accepted speakers will be notified by May 27th. The schedule will be announced on June 3rd.

Birds of a Feather Sessions & Discussion Groups

This year, we will again have space for Birds of a Feather Sessions & Discussion Groups, which are in-depth interactive discussions that allow for collaboration between Xen Project developers and community members. We will publish how you can request a BoF closer to the event. In the meantime, here are the ground rules BoFs:

  • Each BoF host will get 3-5 minutes (depending on the number of BoFs on the day) to pitch your BoF to the entire audience. Slides are not allowed.
  • After we publish the Xen Project Developer schedule, community members that have registered for the summit can submit a request to host a BoF (specifying a couple of slots in preference order)
  • BoFs are small discussion groups, not presentations. You are expected to take notes (or nominate an attendee to do so) and post discussion notes on one of our mailing lists after the summit.

Developer Meeting

I am also pleased to announce that we will also be hosting a 1/2 day Xen Project Developer Meeting the day before the Xen Project Developer Summit (space is limited). The event is open to all members of the Developer Community. More details will follow soon.

Where to stay at the summit

Discounted hotels are listed at the event website at the price of CAD $209.00 per night. Reservations have to be made by July 29th. We are sharing a room block with other Linux Foundation events, so please book early.

Xen Project 4.4.4 Maintenance Release is Available

I am pleased to announce the release of Xen 4.4.4. Xen Project Maintenance releases are released in line with our Maintenance Release Policy. We recommend that all users of the 4.4 stable series update to this point release.

Xen 4.4.4 is available immediately from its git repository:;a=shortlog;h=refs/heads/stable-4.4
    (tag RELEASE-4.4.4)

or from the Xen Project download page at

This release contains many bug fixes and improvements. For a complete list of changes in this release, please check lists of changes on the download page.

Note that this is the last Xen Project coordinated release of the 4.4 stable series. The tree will be switched to security only maintenance mode after this release.