Author Archives: Lars Kurth

About Lars Kurth

Lars Kurth is a highly effective, passionate community manager with strong experience of working with open source communities (Symbian, Symbian DevCo, Eclipse, GNU) and currently is community manager for Lars has 9 years of experience building and leading engineering teams and a track record of executing several change programs impacting 1000 users. Lars has 16 years of industry experience in the tools and mobile sector working at ARM, Symbian Ltd, Symbian Foundation and Nokia. Lars has strong analytical, communication, influencing and presentation skills, good knowledge of marketing and product management and extensive background in C/C , Java and software development practices which he learned working as community manager, product manager, chief architect, engineering manager and software developer. If you want to know more, check out Personally, Lars has a wide range of interests such as literature, theatre, cinema, cooking and gardening. He is particularly fascinated by orchids and carnivorous plants and has built a rather large collection of plants from all over the world. His love for plants extends into a passion for travel, in particular to see plants grow in their native habitats.

Xen Project Hackathon 15 Event Report

After spending almost a week in Shanghai for the Xen Project Hackathon it is time to write up some notes.

More than 48 delegates from Alibaba, Citrix, Desay SV Automotive, GlobalLogic, Fujitsu, Huawei, Intel, Oracle, Suse and Visteon Electronics attended the event, which covered a wide range of topics.

I wanted to thank Susie Li, Hongbo Wang and Mei Yu from Intel for funding and organizing the event.

zizhu_intel Before Registration People Arriving Group Picture


Xen Project Hackathons started originally as pure hackathons, but have over time evolved to follow the Open Space Unconference format, which we tested in 2012 and fully embraced in 2013. It appears to be one of the best formats to foster discussion and problem solving for groups of up to 50 people.

Besides providing an opportunity to meet face-to-face and build bridges, our hackathons have been very successful in tackling difficult issues, which require plenty of interaction. These issues range from modifying our development process and solving architecture problems to conducting difficult design discussions, coordinating inter-dependencies and sharing experiences. Of course we also write code and sometimes conduct live code reviews in smaller groups alongside the discussion sessions.

00019 00028 20150428_161138 20150428_170321

Discussed Topics

At the event, we covered topics such as:

  • Cadence of maintenance releases
  • Numbering of Xen Project Releases
  • Xen 4.6 Release Planning
  • Testing and Testing Frameworks
  • Hot-patching in the Xen Project Hypervisor
  • Changes to the COLO architecture and interdependencies with Migration v2
  • Possible Future Improvements to Live Migration
  • Upstreaming of Intel GVT-g
  • Automotive, including lessons learned on implementing graphics virtualization using OpenGL 2.0 and a walk through of a mediated graphics virtualization solution for the Imagination PowerVR SGX544 GPU on Xen and ARM
  • Xen and OpenStack
  • Evolution of Virtual Machine Introspection (including HW assistance) in the Xen Hypervisor
  • Vendor Strategies For Upgrading Xen in their products (e.g. from Xen 4.1.5 to 4.5)
  • Effectiveness of New Xen Project Security Policy

As usual, we will post summaries (or patches/RFC’s) from these discussions on xen-devel@ – I will also post links to follow-up discussions on our wiki.

Future Xen Project Developer Events in Asia

We’ve learned that the term hackathon is misleading for this event and confuses some of our attendees. Our hackathons are really more of an Architecture Workshop and Design Summit. For this reason, we will probably rename the Hackathon: for a current proposal on the new name check out this and this e-mail thread.

As the event was very successful and we have a growing, active developer community in China, we are considering holding another similar event in 2017 or a Xen Project Developer Summit at LinuxCon Japan in 2017. Stay tuned for more details.

20150427_205419 20150427_205410 20150427_211809 20150429_174059


Introducing the Xen Project – OpenStack CI Loop

We recently introduced the new Xen Project Test Lab, a key piece of infrastructure to improve the quality of our code and code coverage. As stated earlier, “we are serious and proactive when it comes to minimising and, whenever possible, eliminating any adverse effects from defects, security vulnerabilities or performance problems”. This also applies to Xen Project integration with OpenStack, which is why the Xen Project Advisory Board made available funds to create the Xen Project – OpenStack CI Loop in January 2015. We started work on setting up our OpenStack CI Loop immediately after the funds were made available, fixed a number of issues in the Xen Project Hypervisor, Libvirt and OpenStack Nova and are excited to announce that our Xen Project – OpenStack CI Loop is now live and in production.

I wanted to thank members of our Test Infrastructure Working Group, comprised of employees from AMD, Amazon Web Services, Citrix, Intel and Oracle, and community members from Rackspace and Suse who have been supporting the creation of the Xen Project – OpenStack CI Loop.

What does an OpenStack CI Loop Do?

An OpenStack external testing platform (or CI Loop) enables third parties to run tests against an OpenStack environment that is configured with that third party’s drivers or hardware and reports the results of those tests on the code review of a proposed OpenStack patch. It is easy to see the benefit of this real-time feedback by taking a look at a code review that shows how these platforms provide feedback.

In this screenshot, you can see a number Verified +1 and one Verified -1 labels added by CI loops to OpenStack Nova.

In this screenshot, you can see a number Verified +1 and one Verified -1 labels added by CI loops to OpenStack Nova.

The figure below, shows the number of OpenStack Nova drivers for Hypervisors, which allow you to choose which Hypervisor(s) to use for your Nova Deployment. Note that these are classified into groups A, B and C.

This diagram shows the different Nova compute drivers and their quality status

This diagram shows the different Nova compute drivers and their quality status

In a nutshell, the groups have the following meaning:

  • Group C: These drivers have minimal testing.
  • Group B: These drivers have unit tests that gate commits and functional testing providing by an external system that does not gate commits, but advises patch authors and reviewers of results in the OpenStack code review system.
  • Group A: These drivers have unit tests that gate commits and functional testing that gate commits.

With the introduction of the Xen Project – OpenStack CI Loop we are close to achieving our first goal of moving the Xen Project Hypervisor from Group C to B. This is a goal we first publicly stated at this year’s FOSDEM’15 where some of you may had had the opportunity to hear Stefano Stabellini talk about using the Xen Project Hypervisor in OpenStack.

What we test against?

Currently, our CI Loop tests all OpenStack Nova commits against Ubuntu Xen 4.4.1 packages with a number of patches applied to them, together Libvirt 1.2.14 with a number of patches applied to them (for more details see this specification). All the patches are already integrated upstream. When off-the shelf packages of these changes are available in Linux distros, we will start testing against them.

What is next?

Monitor the CI loop test results against the official OpenStack CI: we will run the CI loop for a few weeks and monitor, whether there are any discrepancies between the results of the official Nova CI loop and ours. If there are any, we will investigate where the issue is and fix them. This is important, as we had a number of intermittent issues in the Xen Libvirt driver and we want to be sure, we have fixed them all.

Fix two known issues, and enable two disabled tests: We have two test cases related to iSCSI support (test_volume_boot_pattern), which are currently disabled. In one case, we believe the that the Tempest Integration Suite makes some KVM specific assumptions: Xen Project community members are working with the OpenStack community to fix the issue. The second issue is related, but requires further investigation.

Make the Xen Project – OpenStack CI Loop voting: The next logical step, is to work with the OpenStack community to make our CI loop voting. This is the first step to move from group C to B.

Beyond That: Build trust and become an active member of the OpenStack community. Our ultimate goal, is to ensure that the Xen Project Hypervisor moves into group A, alongside KVM. We are also investigating, whether it makes sense to run Xen Project + Libvirt against OpenStack Neutron. We are also looking at integrating OpenStack Tempest into our new Xen Project Test Lab, such that we can ensure that upstream Xen and Libvirt will work with OpenStack Nova.

Can I Help?

If you use Xen Project with OpenStack and Libvirt, feel free to get in touch. Some members of the Xen Project community will be at the Vancouver OpenStack Summit and we are keen to learn, what issues you have and whether we can help fixing them. As community manager, will help arrange meetings and build bridges: feel free to contact me under community dot manager at xenproject dot org. You can also ask questions on xen-devel@ and xen-users@ and connect with Xen Project Developers and Users.

Additional Resources

Introducing Xen Project’s New Test Lab

One of Xen Project’s highest priorities is to continually work to improve the quality of our code and code coverage. We’re serious and proactive when it comes to minimising and, whenever possible, eliminating any adverse effects from defects, security vulnerabilities or performance problems. Our users run some of the largest cloud and datacenter operations in the world, so we know that reboots and service interruptions are more than mere glitches in operations.

That’s why the Xen Project Advisory Board has made a significant investment to set up a Xen Project-owned Test Lab based on OSSTEST. We’re excited to announce that the lab is now live and in production.

Our new test lab will allows us to improve upstream quality of the Xen Project Hypervisor, while developers from different organizations now have access to a system that can be used to investigate test failures, schedule test jobs, etc. The test lab is also being used for regression testing of the Xen Project Hypervisor against different upstream and downstream environments. The Xen Project also tests against upstream and downstream open source projects such as the Linux Kernel and OpenStack to ensure that the Xen Project Hypervisor works seamlessly with important key technologies.

To help kick-start the lab, we created the Test Infrastructure Working Group, which helped source a COLO, spec out the hardware requirements for the test system and helped procure the machines needed to run OSSTEST. A special thank you goes to Ian Jackson, who has been instrumental in getting this effort off the ground over the past six months. We also appreciate contributions from our committee comprised of employees from AMD, Amazon Web Services, Citrix, Intel and Oracle, who have worked closely with our maintainers, project leads and wider community to create the new testing facility. As a result, we’ve decommissioned the old system run by Citrix as a service to the Xen Project community.

In this blog, we’ll describe our platform in more detail, identify what’ we’re doing today with OSSTEST and new goals from our Test Infrastructure Working Group.

Xen Project’s New Test Lab

This picture shows a section of our test lab, which is co-located at Earthlink

This picture shows a section of our test lab, which is co-located at the Earthlink Boston Data Centre

Currently the Xen Project owns a rack with a mixture of 24 x86 Intel and AMD hosts, and 8 ARM hosts (a custom-made set-up that includes 4 Cubietruck and 4 Arndale boards). Although we are currently only at 50 percent of planned capacity, the new test system has a much more diverse set of test machines and already offers more capacity than the old one did. It is also significantly more reliable than the old system.

Setting up a test lab with several different architectures and suppliers as well as a globally dispersed community is simply a challenge. For example, we discovered a number of bugs in the Linux kernel, FreeBSD and Hypervisor, which are currently being addressed. We also tripped over a number of unexpected issues, such as hardware issues and BIOS bugs in some of the machines in the COLO.

How Do We Use OSSTEST Today?

As mentioned earlier, the test lab is being used for regression testing. We’re testing different versions of Xen Project Hypervisor against different versions of the Linux Kernel, Linux distros, FreeBSD, Windows, QEMU, Rump Kernels and other open source projects we interface with. We also do test specific features such as Xen Security Modules, Credit 2 scheduler and other Xen Project features. And more recently we added functionality for performance testing of the Xen Project Hypervisor.

The set-up is essentially following a continuous integration paradigm; changes to Xen Project trees are pushed onto a staging branch. When tests succeed, changes are automatically pushed to the master branch. If there are failures, a bisection algorithm is run, which in many cases will identify the changeset that is causing the issue and developers are then notified that a specific changeset has created a problem.

This figure shows how the interaction between the test lab, the staging branch and the master branch.

This figure shows how the contribution workflow and its interaction between the test lab, the staging branch and the master branch.

What’s Next for Xen Project’s Test Lab

Bring the test lab to 100% capacity. As we work through the remaining issues, new machines will be taken into production as issues are resolved. The increased capacity will enable us to add on-demand testing capability, which will enable community members to test on hardware that they may not have themselves.

Broaden Access: The new system is in a community-operated COLO, rather than one hosted inside the Citrix private network. This, and the increased capacity, will mean that we can to grant access to the facility to Xen Project community members regardless of their affiliation, and provide an on-demand testing capability. To do this, we will need to agree upon processes to enable community members to access the test lab, in a similar way as we allow access to Coverity Scan data.

More Test Coverage: This increased capacity will allow us to further increase the breadth and depth of our testing. We have a number of enhancements in the pipeline, including for example testing of FreeBSD hosts, nested virtualisation, and compatibility testing with a variety of OS distributions.

Expand the infrastructure and add another rack. The Xen Project Advisory Board agreed to fund another rack and to procure further test machines in 2015. We expect that some 64-bit ARM machines will be part of the next tranche of machines that the Xen Project will buy.

Xen Project OpenStack CI loop: In addition to the Xen Project Test Lab, the Advisory Board is also funding a 3rd party continuous integration loop to improve Xen Project virtualization and OpenStack integration. A prototype of the CI loop is already running at Xen Project community members are currently fixing a number of issues in OpenStack Nova, Libvirt and Xen Project, such that the CI loop can become voting on OpenStack Nova commits.

We also continue to run the Coverity Scan static analysis service, which helps us quickly find, review and resolve Xen Project Hypervisor code defects and vulnerabilities. We’ve done this since 2013 and continue to regularly review and fix reported bugs.

If you’re interested in testing Xen, we encourage developers to join the xen-devel mailing list and ask how you can help. We’re always looking for help extending the set of test cases to tune performance for as wide a set of configurations and features as possible.

Xen Project Participates in Outreachy (formerly OPW)

This is a quick reminder that the Xen Project is again participating in Outreachy (formerly known as OPW/Outreach program for Women).

Outreachy is the successor of the Outreach Program for Women. Please see its page for the information about the May 2015 round of interships.

Outreach Program for Women has been helping women (cis and trans), trans men, and genderqueer people get involved in free and open source software. It provided a supportive community for beginning to contribute any time throughout the year and offered focused internship opportunities twice a year with a number of free software organizations. This work is continued by Outreachy, with the goal of expanding the program to more participants from underrepresented backgrounds.

Round 10 of Outreachy

outreachy-poster-2015-May-AugustThe application deadline for interns is March 24, 2015 April 7, 2015. For a list of projects for interns and more information on how to apply, check our Xen Project Outreachy portal.

We have many different projects in many different areas! Check out the following project ideas: Outreach Program Projects, General List of Development Projects or MirageOS Pioneer Projects.

Learn about the Experience of past Applicants

At least year’s Xen Project Developer Summit, we ran a panel discussion that included OPW interns, GSoC students as well as mentors.

You may also want to read Women interns rocking open source at Xen Project.

GSoC 2015

Unfortunately the project has not been accepted for GSoC in 2015! However, there are several Xen related projects hosted in various mentoring organisations, such as

Please apply to these projects using the normal GSoC application process.

Intel hosts Xen Project Hackathon, April 28-29 in Shanghai

I am pleased to announce the next Xen Project Hackathon to be held this spring.  Although we call it a Hackathon, the event consists of several parallel sessions in which Xen Project developers will create, discuss and review designs and changes that impact Xen’s architecture. We’ll perform code reviews, discuss our future roadmap, work on improving the development process, tackle debug problems in the code base and cover other development related topics. Sessions are very interactive: typically there are no presentations.

Intel-logoThe Hackathon will be hosted by Intel at their Shanghai Zizhu Campus, April 28-29. I wanted to thank Susie Li and Mei Yu from Intel for hosting the Hackathon. Intel has been one of the core contributors to the Xen Project since 2003 and has been contributing many features to the Project. Intel joined the Xen Project Advisory Board in 2013 when the software became a Linux Foundation Collaborative Project. We recently interviewed Donald Dugger, Intel’s Virtualization Architect, to find out why Intel continues to support, contribute and invest in the Xen Project.

What to expect at a Xen Project Hackathon?

The aim of the Hackathon is to give developers the opportunity to meet face-to-face to discuss development, coordinate, write code and collaborate with other developers. Of course, the event will allow everyone to meet in-person and build relationships; to facilitate this, we will have a social event on the evening of the 28th. We will cover many hot topics such as the latest Xen Project Hypervisor 4.6 features, planning for the next Xen Project Hypervisor release, Cloud Integration, Cloud Operating Systems, MirageOS, as well as new opportunities in embedded, mobile, automotive and NFV. But at the end of the day, the community chooses what topics will be covered.

To ensure that the event runs efficiently, each day is divided into several segments. We will have a number of work areas that are labelled with numbers (or other unique identifiers). Each morning will start with a plenary and scheduling session. Every attendee can propose a session, which we will map against a work area and time-slot. This makes it easy for other attendees to participate in projects and sessions they care about. Of course we also encourage attendees to highlight projects they plan to share before the event by adding them to our wiki.

How to Register

Spaces for the Xen Project Hackathon are limited (we can accommodate 50 people). Be sure to request an invitation to the event before our cut-off registration date of April 12th, 2015.

More Information

Updates to Xen Project Security Process

lockBefore Christmas, the Xen Project ran a community consultation to refine its Security Problem Response Process.  We recently approved changes that, in essence, are tweaks to our existing process, which is based on a Responsible Disclosure philosophy.

Responsible Disclosure and our Security Problem Response Process are important components of keeping users of Xen Project-based products and services safe from security exploits. Both ensure that products and services can be patched by members of the pre-disclosure list before details of a vulnerability are published and before said vulnerabilities can be exploited by black hats.

The changes to our response process fall into a number of categories:

  • Clarify whether security updates can be deployed on publicly hosted systems (e.g. cloud or hosting providers) during embargo
  • Sharing of information among pre-disclosure list members
  • Applications procedure for pre-disclosure list membership

The complete discussion leading to the changes, the concrete changes to the process, and the voting records supporting the changes are tracked in Bug #44 -Security policy ambiguities. On February 11, 2015, the proposed changes were approved in accordance with Xen Project governance. Note that some process changes are already implemented, whereas others are waiting for implementation tasks (e.g. new secure mailing lists) before they can fully be put in place. We have however updated our Security Problem Response Process as the most important elements of the process are already in place.

Process Changes Already in Operation

The updated policy makes explicit whether or not patches related to a Xen Security Issue can be deployed by pre-disclosure list members. The concrete policy changes can be found here and here. In practice, every Xen Security Advisory will contain a section such as:


Deployment of the patches and/or mitigations described above (or
others which are substantially similar) is permitted during the
embargo, even on public-facing systems with untrusted guest users and

Predisclosure list members who wish to deploy significantly different
patches and/or mitigations, please contact the Xen Project Security

This section will clarify whether deploying fixed versions of Xen during the embargo is allowed. Any restrictions will also be stated in the embargoed advisory. The Security Team will impose deployment restrictions only to prevent the exposure of security vulnerability technicalities, which present a significant risk of vulnerability rediscovery (for example, by visible differences in behaviour). Such situations have been, and are expected, to be rare.

Changes to Application Procedure for Pre-disclosure List Membership

We also made additional changes related to streamlining and simplifying the process of applying for pre-disclosure list membership. Detailed policy changes can be found here and here. Moving forward, future applications to become members of the Xen Project pre-disclosure list have to be made publicly on the predisclosure-applications mailing list. This enables Xen Project community members to provide additional information and also is in line with one of our community’s core principles: transparency. In addition, we’ve clarified our eligibility criteria to make it easier for the Xen Project Security Team, as well as observers of the mailing list, to verify whether applicants are eligible to become members of the list.

Process Changes Not Fully Implemented:  Sharing of Information Among Pre-disclosure List Members

Finally, members of the pre-disclosure list will be explicitly allowed to share fixes to embargoed issues, analysis, and other relevant information with the security teams of other pre-disclosure members. Information sharing will happen on a private and secure mailing list hosted by the Xen Project.  Note the list is not yet in place; more details here.