Author Archives: Lars Kurth

About Lars Kurth

Lars Kurth is a highly effective, passionate community manager with strong experience of working with open source communities (Symbian, Symbian DevCo, Eclipse, GNU) and currently is community manager for xen.org. Lars has 9 years of experience building and leading engineering teams and a track record of executing several change programs impacting 1000 users. Lars has 16 years of industry experience in the tools and mobile sector working at ARM, Symbian Ltd, Symbian Foundation and Nokia. Lars has strong analytical, communication, influencing and presentation skills, good knowledge of marketing and product management and extensive background in C/C , Java and software development practices which he learned working as community manager, product manager, chief architect, engineering manager and software developer. If you want to know more, check out uk.linkedin.com/in/larskurth. Personally, Lars has a wide range of interests such as literature, theatre, cinema, cooking and gardening. He is particularly fascinated by orchids and carnivorous plants and has built a rather large collection of plants from all over the world. His love for plants extends into a passion for travel, in particular to see plants grow in their native habitats.

Xen Project 4.4.3 Maintenance Release is Available

I am pleased to announce the release of Xen 4.4.3. We recommend that all users of the 4.4 stable series update to this latest maintenance release.

Xen 4.4.3 is available immediately from its git repository:

    xenbits.xenproject.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.4
    (tag RELEASE-4.4.3)

or from the Xen Project download page at www.xenproject.org/downloads/xen-archives/xen-44-series/xen-443.html.

This release contains many bug fixes and improvements. For example:

  • A number of bug fixes that affected libvirt, in particular when used with OpenStack (this release contains all changes that we use in the Xen Project OpenStack CI loop; also see related OpenStack news, Latest News on Xen Support in libvirt & Xen and OpenStack);
  • Stability improvements to CPUPOOL handling, in particular when used with different schedulers;
  • Stability improvements to EFI support on some x86 platforms;
  • Security fixes since the release of Xen 4.4.2

For a complete list of changes in this release, please check lists of changes on the download page.

Xen Project 4.5.1 Maintenance Release Available

I am pleased to announce the release of Xen 4.5.1. We recommend that all users of the 4.5 stable series update to this first point release.

Xen 4.5.1 is available immediately from its git repository:

    xenbits.xenproject.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.5
    (tag RELEASE-4.5.1)

or from the Xen Project download page at www.xenproject.org/downloads/xen-archives/xen-45-series/xen-451.html.

This release contains many bug fixes and improvements. For example:

  • Removal of race conditions in the Xen default toolstack that affected libvirt, in particular when used with OpenStack (this release contains all changes that we use in the Xen Project OpenStack CI loop; also see related OpenStack news);
  • Stability improvements to CPUPOOL handling, in particular when used with different schedulers;
  • Stability improvements to EFI support on some x86 platforms;
  • Stability improvement to handling of nested virtualisation on x86;
  • Various improvements to 32 and 64 bit ARM support;
  • Various improvements to better integrate and support rump kernels;
  • Error handling improvements; and
  • Security fixes since the release of Xen 4.5.0,

For a complete list of changes in this release, please check lists of changes on the download page.

2015 Xen Project Developer Summit Line-up Announced

I am pleased to announce the schedule for the Xen Project Developer Summit. The event will take place in Seattle on August 17-18, 2015.

The Xen Project Developer Summit brings together its community of developers and power users. Each year the event features the latest developments, best practices, collaboration, product roadmap updates and future planning from developers who are leading the way in server density, million-node data centers, automotive, mobile, graphic-intensive workloads, cloud and enterprise security.

For the first time, Xen Project Developer Summit and KVM Forum will co-host a Hackathon on Aug. 18 aimed at fostering technical collaboration between the two leading open source hypervisors in IT today. KVM and Xen users and developers will have the opportunity to collaborate and delve into work on libvirt code. A co-hosted evening event will be held that night.

Following is a sampling of confirmed speakers and presentations to be discussed in Seattle:

  • Dario Faggioli, senior software engineer, Citrix, and Meng Xu, PhD Student, University of Pennsylvania, will co-present the state of scheduling in Xen and the hypervisor provides a set of schedulers, each one suited for specific use cases.
  • Sainath Grandhi, Core OS engineer, Intel will present Xen containers to run Docker container applications sandboxed in a small VM as an alternative to bare-metal containers, providing tighter security and resource isolation.
  • Julien Grall, software engineer, Citrix, will target developers making a product based on Xen ARM requiring device assignment.
  • Juergen Gross, Linux kernel developer, SUSE, will outline suggestions for a systems architecture that is much more fault tolerant against hardware and software failures.
  • Manish Jaggi, Xen/KVM hypervisor technical lead, Cavium, will talk about Xen support for ThunderX, a family of highly integrated, multi-core SoC processors based on 64-bit ARMv8 architecture, for data center and cloud applications.
  • Liu Jinsong, PM and RAS maintainer, Alibaba, will introduce live migration at AliCloud, summarizing the technical virtualization, network, and storage problems that need to be solved to make live migration work at AliCloud.
  • Mark Kraeling, product manager, GE Transportation, will talk about virtualizing the locomotive and how GE uses about how GE uses Xen for x86-based processors, and KVM for ARM-based processors.
  • Tamas Lengyel, security researcher at Novetta, University of Connecticut PhD student, will shed more light on current trends in virtualization security, pitfalls and new features coming in 4.6.
  • Wei Liu, Xen 4.6 release manager, Citrix, will give a status report on the upcoming Xen Project 4.6 release.
    Stefano Stabellini, senior principle software engineer, Citrix, will explain how to deploy OpenStack using the Xen Project hypervisor to run your VMs.
  • Zhi Wang, engineer, Intel, will provide a detailed update on the evolution of Intel Graphics Virtualization Technology for full GPU virtualization.
  • Konrad Wilk, software director, Oracle, will discuss the design and functionality of xSplice, which offers a method for live patching without requiring a system to reboot.
  • Marc Zyngier, kernel hacker, ARM, will share a hypervisor agnostic view of virtualization extensions added to the latest ARM architecture.

Birds of a Feather session and Discussions

Besides presentations, the developer summit will also provide an opportunity for in-depth interactive discussions (Birds of a Feather sessions), which allow deep interaction and collaboration between Xen Project developers and community members. These will happen in a second track alongside part of the main event. To submit a BoF, please send an email to community dot manager at xenproject dot org and chose a BoF slot, title and short description or fill out this wiki page.

Registration

For more information about Xen Project Developer Summit 2015, including how to register and to view the complete schedule, visit: events.linuxfoundation.org/events/xen-project-developer-summit.

Xen Project now in OpenStack Nova Hypervisor Driver Quality Group B

A few weeks ago, we introduced the Xen Project – OpenStack CI Loop, which is testing Nova commits against the Xen Project Hypervisor and Libvirt. Xen Project community is pleased to announce that we have moved from Quality Group C to B, as we’ve made significant progress in the last few weeks and the Xen Project CI loop is now voting on Nova commits.

This diagram shows the number of OpenStack Nova drivers for Hypervisors, which allow you to choose which Hypervisor(s) to use for your Nova Deployment. Note that these are classified into groups A, B and C. Xen Project is now in Quality Group B.

This diagram shows the number of OpenStack Nova drivers for Hypervisors, which allow you to choose which Hypervisor(s) to use for your Nova Deployment. Note that these are classified into groups A, B and C. Xen Project is now in Quality Group B.

Quality groups are defined as follows:

  • Group C: These drivers have minimal testing and may or may not work at any given time. Test coverage may include unit tests that gate commits. There is no public functional testing.
  • Group B: Test coverage includes unit tests that gate commits. Functional testing is provided by an external system (such as our CI loop) that do not gate commits, but advises patch authors and reviewers of results in OpenStack Gerrit.
  • Group A: Test coverage includes unit tests and functional testing that both gate commits.

What does this mean in practice?

The easiest way to understand what this means in practice, is to look at a real code review as shown in the figure below.

This diagram shows how the functional tests for the Xen Project initially failed, and passed after a new patch was uploaded.

This diagram shows how the functional tests for the Xen Project initially failed, and passed after a new patch was uploaded.

This code review shows that the OpenStack Jenkins instance (running on KVM) and the Xen Project CI Loop failed their respective tests initially, so a new patchset was uploaded and the tests succeeded afterwards.

This diagram shows the status after a new patchset was uploaded. Note that the review of the patchset is still pending manual review.

This diagram shows the status after a new patchset was uploaded. Note that the review of the patchset is still pending manual review.

Also see Merging.Repository Gating in the OpenStack documentation.

Relevant OpenStack Summit Sessions

There are a number of sessions at this week’s OpenStack Summit that are worth attending including:

Hands-on sessions to improve 3rd party CI infrastructure include:

Other third-party CI-related sessions include:

Relevant Regular Meetings

Note that there are also weekly Third Party CI Working Group meetings for all operators of 3rd party CI loops in #openstack-meeting-4 on Wednesdays at 1500/0400 UTC alternating organized by Kurt Taylor (krtaylor). Third party CI operators interested in enhancing documentation, reviewing patches for relevant work, and improving the consumability of infra CI components are encouraged to attend. See here for more information on the working group.

Xen Project Hackathon 15 Event Report

After spending almost a week in Shanghai for the Xen Project Hackathon it is time to write up some notes.

More than 48 delegates from Alibaba, Citrix, Desay SV Automotive, GlobalLogic, Fujitsu, Huawei, Intel, Oracle, Suse and Visteon Electronics attended the event, which covered a wide range of topics.

I wanted to thank Susie Li, Hongbo Wang and Mei Yu from Intel for funding and organizing the event.

zizhu_intel Before Registration People Arriving Group Picture

Format

Xen Project Hackathons started originally as pure hackathons, but have over time evolved to follow the Open Space Unconference format, which we tested in 2012 and fully embraced in 2013. It appears to be one of the best formats to foster discussion and problem solving for groups of up to 50 people.

Besides providing an opportunity to meet face-to-face and build bridges, our hackathons have been very successful in tackling difficult issues, which require plenty of interaction. These issues range from modifying our development process and solving architecture problems to conducting difficult design discussions, coordinating inter-dependencies and sharing experiences. Of course we also write code and sometimes conduct live code reviews in smaller groups alongside the discussion sessions.

00019 00028 20150428_161138 20150428_170321

Discussed Topics

At the event, we covered topics such as:

  • Cadence of maintenance releases
  • Numbering of Xen Project Releases
  • Xen 4.6 Release Planning
  • Testing and Testing Frameworks
  • Hot-patching in the Xen Project Hypervisor
  • Changes to the COLO architecture and interdependencies with Migration v2
  • Possible Future Improvements to Live Migration
  • Upstreaming of Intel GVT-g
  • Automotive, including lessons learned on implementing graphics virtualization using OpenGL 2.0 and a walk through of a mediated graphics virtualization solution for the Imagination PowerVR SGX544 GPU on Xen and ARM
  • Xen and OpenStack
  • Evolution of Virtual Machine Introspection (including HW assistance) in the Xen Hypervisor
  • Vendor Strategies For Upgrading Xen in their products (e.g. from Xen 4.1.5 to 4.5)
  • Effectiveness of New Xen Project Security Policy

As usual, we will post summaries (or patches/RFC’s) from these discussions on xen-devel@ – I will also post links to follow-up discussions on our wiki.

Future Xen Project Developer Events in Asia

We’ve learned that the term hackathon is misleading for this event and confuses some of our attendees. Our hackathons are really more of an Architecture Workshop and Design Summit. For this reason, we will probably rename the Hackathon: for a current proposal on the new name check out this and this e-mail thread.

As the event was very successful and we have a growing, active developer community in China, we are considering holding another similar event in 2017 or a Xen Project Developer Summit at LinuxCon Japan in 2017. Stay tuned for more details.

20150427_205419 20150427_205410 20150427_211809 20150429_174059

 

Introducing the Xen Project – OpenStack CI Loop

We recently introduced the new Xen Project Test Lab, a key piece of infrastructure to improve the quality of our code and code coverage. As stated earlier, “we are serious and proactive when it comes to minimising and, whenever possible, eliminating any adverse effects from defects, security vulnerabilities or performance problems”. This also applies to Xen Project integration with OpenStack, which is why the Xen Project Advisory Board made available funds to create the Xen Project – OpenStack CI Loop in January 2015. We started work on setting up our OpenStack CI Loop immediately after the funds were made available, fixed a number of issues in the Xen Project Hypervisor, Libvirt and OpenStack Nova and are excited to announce that our Xen Project – OpenStack CI Loop is now live and in production.

I wanted to thank members of our Test Infrastructure Working Group, comprised of employees from AMD, Amazon Web Services, Citrix, Intel and Oracle, and community members from Rackspace and Suse who have been supporting the creation of the Xen Project – OpenStack CI Loop.

What does an OpenStack CI Loop Do?

An OpenStack external testing platform (or CI Loop) enables third parties to run tests against an OpenStack environment that is configured with that third party’s drivers or hardware and reports the results of those tests on the code review of a proposed OpenStack patch. It is easy to see the benefit of this real-time feedback by taking a look at a code review that shows how these platforms provide feedback.

In this screenshot, you can see a number Verified +1 and one Verified -1 labels added by CI loops to OpenStack Nova.

In this screenshot, you can see a number Verified +1 and one Verified -1 labels added by CI loops to OpenStack Nova.

The figure below, shows the number of OpenStack Nova drivers for Hypervisors, which allow you to choose which Hypervisor(s) to use for your Nova Deployment. Note that these are classified into groups A, B and C.

This diagram shows the different Nova compute drivers and their quality status

This diagram shows the different Nova compute drivers and their quality status

In a nutshell, the groups have the following meaning:

  • Group C: These drivers have minimal testing.
  • Group B: These drivers have unit tests that gate commits and functional testing providing by an external system that does not gate commits, but advises patch authors and reviewers of results in the OpenStack code review system.
  • Group A: These drivers have unit tests that gate commits and functional testing that gate commits.

With the introduction of the Xen Project – OpenStack CI Loop we are close to achieving our first goal of moving the Xen Project Hypervisor from Group C to B. This is a goal we first publicly stated at this year’s FOSDEM’15 where some of you may had had the opportunity to hear Stefano Stabellini talk about using the Xen Project Hypervisor in OpenStack.

What we test against?

Currently, our CI Loop tests all OpenStack Nova commits against Ubuntu Xen 4.4.1 packages with a number of patches applied to them, together Libvirt 1.2.14 with a number of patches applied to them (for more details see this specification). All the patches are already integrated upstream. When off-the shelf packages of these changes are available in Linux distros, we will start testing against them.

What is next?

Monitor the CI loop test results against the official OpenStack CI: we will run the CI loop for a few weeks and monitor, whether there are any discrepancies between the results of the official Nova CI loop and ours. If there are any, we will investigate where the issue is and fix them. This is important, as we had a number of intermittent issues in the Xen Libvirt driver and we want to be sure, we have fixed them all.

Fix two known issues, and enable two disabled tests: We have two test cases related to iSCSI support (test_volume_boot_pattern), which are currently disabled. In one case, we believe the that the Tempest Integration Suite makes some KVM specific assumptions: Xen Project community members are working with the OpenStack community to fix the issue. The second issue is related, but requires further investigation.

Make the Xen Project – OpenStack CI Loop voting: The next logical step, is to work with the OpenStack community to make our CI loop voting. This is the first step to move from group C to B.

Beyond That: Build trust and become an active member of the OpenStack community. Our ultimate goal, is to ensure that the Xen Project Hypervisor moves into group A, alongside KVM. We are also investigating, whether it makes sense to run Xen Project + Libvirt against OpenStack Neutron. We are also looking at integrating OpenStack Tempest into our new Xen Project Test Lab, such that we can ensure that upstream Xen and Libvirt will work with OpenStack Nova.

Can I Help?

If you use Xen Project with OpenStack and Libvirt, feel free to get in touch. Some members of the Xen Project community will be at the Vancouver OpenStack Summit and we are keen to learn, what issues you have and whether we can help fixing them. As community manager, will help arrange meetings and build bridges: feel free to contact me under community dot manager at xenproject dot org. You can also ask questions on xen-devel@ and xen-users@ and connect with Xen Project Developers and Users.

Additional Resources