Author Archives: Lars Kurth

About Lars Kurth

Lars Kurth is a highly effective, passionate community manager with strong experience of working with open source communities (Symbian, Symbian DevCo, Eclipse, GNU) and currently is community manager for xen.org. Lars has 9 years of experience building and leading engineering teams and a track record of executing several change programs impacting 1000 users. Lars has 16 years of industry experience in the tools and mobile sector working at ARM, Symbian Ltd, Symbian Foundation and Nokia. Lars has strong analytical, communication, influencing and presentation skills, good knowledge of marketing and product management and extensive background in C/C , Java and software development practices which he learned working as community manager, product manager, chief architect, engineering manager and software developer. If you want to know more, check out uk.linkedin.com/in/larskurth. Personally, Lars has a wide range of interests such as literature, theatre, cinema, cooking and gardening. He is particularly fascinated by orchids and carnivorous plants and has built a rather large collection of plants from all over the world. His love for plants extends into a passion for travel, in particular to see plants grow in their native habitats.

2015 Xen Project Developer Summit Line-up Announced

I am pleased to announce the schedule for the Xen Project Developer Summit. The event will take place in Seattle on August 17-18, 2015.

The Xen Project Developer Summit brings together its community of developers and power users. Each year the event features the latest developments, best practices, collaboration, product roadmap updates and future planning from developers who are leading the way in server density, million-node data centers, automotive, mobile, graphic-intensive workloads, cloud and enterprise security.

For the first time, Xen Project Developer Summit and KVM Forum will co-host a Hackathon on Aug. 18 aimed at fostering technical collaboration between the two leading open source hypervisors in IT today. KVM and Xen users and developers will have the opportunity to collaborate and delve into work on libvirt code. A co-hosted evening event will be held that night.

Following is a sampling of confirmed speakers and presentations to be discussed in Seattle:

  • Dario Faggioli, senior software engineer, Citrix, and Meng Xu, PhD Student, University of Pennsylvania, will co-present the state of scheduling in Xen and the hypervisor provides a set of schedulers, each one suited for specific use cases.
  • Sainath Grandhi, Core OS engineer, Intel will present Xen containers to run Docker container applications sandboxed in a small VM as an alternative to bare-metal containers, providing tighter security and resource isolation.
  • Julien Grall, software engineer, Citrix, will target developers making a product based on Xen ARM requiring device assignment.
  • Juergen Gross, Linux kernel developer, SUSE, will outline suggestions for a systems architecture that is much more fault tolerant against hardware and software failures.
  • Manish Jaggi, Xen/KVM hypervisor technical lead, Cavium, will talk about Xen support for ThunderX, a family of highly integrated, multi-core SoC processors based on 64-bit ARMv8 architecture, for data center and cloud applications.
  • Liu Jinsong, PM and RAS maintainer, Alibaba, will introduce live migration at AliCloud, summarizing the technical virtualization, network, and storage problems that need to be solved to make live migration work at AliCloud.
  • Mark Kraeling, product manager, GE Transportation, will talk about virtualizing the locomotive and how GE uses about how GE uses Xen for x86-based processors, and KVM for ARM-based processors.
  • Tamas Lengyel, security researcher at Novetta, University of Connecticut PhD student, will shed more light on current trends in virtualization security, pitfalls and new features coming in 4.6.
  • Wei Liu, Xen 4.6 release manager, Citrix, will give a status report on the upcoming Xen Project 4.6 release.
    Stefano Stabellini, senior principle software engineer, Citrix, will explain how to deploy OpenStack using the Xen Project hypervisor to run your VMs.
  • Zhi Wang, engineer, Intel, will provide a detailed update on the evolution of Intel Graphics Virtualization Technology for full GPU virtualization.
  • Konrad Wilk, software director, Oracle, will discuss the design and functionality of xSplice, which offers a method for live patching without requiring a system to reboot.
  • Marc Zyngier, kernel hacker, ARM, will share a hypervisor agnostic view of virtualization extensions added to the latest ARM architecture.

Birds of a Feather session and Discussions

Besides presentations, the developer summit will also provide an opportunity for in-depth interactive discussions (Birds of a Feather sessions), which allow deep interaction and collaboration between Xen Project developers and community members. These will happen in a second track alongside part of the main event. To submit a BoF, please send an email to community dot manager at xenproject dot org and chose a BoF slot, title and short description or fill out this wiki page.

Registration

For more information about Xen Project Developer Summit 2015, including how to register and to view the complete schedule, visit: events.linuxfoundation.org/events/xen-project-developer-summit.

Xen Project now in OpenStack Nova Hypervisor Driver Quality Group B

A few weeks ago, we introduced the Xen Project – OpenStack CI Loop, which is testing Nova commits against the Xen Project Hypervisor and Libvirt. Xen Project community is pleased to announce that we have moved from Quality Group C to B, as we’ve made significant progress in the last few weeks and the Xen Project CI loop is now voting on Nova commits.

This diagram shows the number of OpenStack Nova drivers for Hypervisors, which allow you to choose which Hypervisor(s) to use for your Nova Deployment. Note that these are classified into groups A, B and C. Xen Project is now in Quality Group B.

This diagram shows the number of OpenStack Nova drivers for Hypervisors, which allow you to choose which Hypervisor(s) to use for your Nova Deployment. Note that these are classified into groups A, B and C. Xen Project is now in Quality Group B.

Quality groups are defined as follows:

  • Group C: These drivers have minimal testing and may or may not work at any given time. Test coverage may include unit tests that gate commits. There is no public functional testing.
  • Group B: Test coverage includes unit tests that gate commits. Functional testing is provided by an external system (such as our CI loop) that do not gate commits, but advises patch authors and reviewers of results in OpenStack Gerrit.
  • Group A: Test coverage includes unit tests and functional testing that both gate commits.

What does this mean in practice?

The easiest way to understand what this means in practice, is to look at a real code review as shown in the figure below.

This diagram shows how the functional tests for the Xen Project initially failed, and passed after a new patch was uploaded.

This diagram shows how the functional tests for the Xen Project initially failed, and passed after a new patch was uploaded.

This code review shows that the OpenStack Jenkins instance (running on KVM) and the Xen Project CI Loop failed their respective tests initially, so a new patchset was uploaded and the tests succeeded afterwards.

This diagram shows the status after a new patchset was uploaded. Note that the review of the patchset is still pending manual review.

This diagram shows the status after a new patchset was uploaded. Note that the review of the patchset is still pending manual review.

Also see Merging.Repository Gating in the OpenStack documentation.

Relevant OpenStack Summit Sessions

There are a number of sessions at this week’s OpenStack Summit that are worth attending including:

Hands-on sessions to improve 3rd party CI infrastructure include:

Other third-party CI-related sessions include:

Relevant Regular Meetings

Note that there are also weekly Third Party CI Working Group meetings for all operators of 3rd party CI loops in #openstack-meeting-4 on Wednesdays at 1500/0400 UTC alternating organized by Kurt Taylor (krtaylor). Third party CI operators interested in enhancing documentation, reviewing patches for relevant work, and improving the consumability of infra CI components are encouraged to attend. See here for more information on the working group.

Xen Project Hackathon 15 Event Report

After spending almost a week in Shanghai for the Xen Project Hackathon it is time to write up some notes.

More than 48 delegates from Alibaba, Citrix, Desay SV Automotive, GlobalLogic, Fujitsu, Huawei, Intel, Oracle, Suse and Visteon Electronics attended the event, which covered a wide range of topics.

I wanted to thank Susie Li, Hongbo Wang and Mei Yu from Intel for funding and organizing the event.

zizhu_intel Before Registration People Arriving Group Picture

Format

Xen Project Hackathons started originally as pure hackathons, but have over time evolved to follow the Open Space Unconference format, which we tested in 2012 and fully embraced in 2013. It appears to be one of the best formats to foster discussion and problem solving for groups of up to 50 people.

Besides providing an opportunity to meet face-to-face and build bridges, our hackathons have been very successful in tackling difficult issues, which require plenty of interaction. These issues range from modifying our development process and solving architecture problems to conducting difficult design discussions, coordinating inter-dependencies and sharing experiences. Of course we also write code and sometimes conduct live code reviews in smaller groups alongside the discussion sessions.

00019 00028 20150428_161138 20150428_170321

Discussed Topics

At the event, we covered topics such as:

  • Cadence of maintenance releases
  • Numbering of Xen Project Releases
  • Xen 4.6 Release Planning
  • Testing and Testing Frameworks
  • Hot-patching in the Xen Project Hypervisor
  • Changes to the COLO architecture and interdependencies with Migration v2
  • Possible Future Improvements to Live Migration
  • Upstreaming of Intel GVT-g
  • Automotive, including lessons learned on implementing graphics virtualization using OpenGL 2.0 and a walk through of a mediated graphics virtualization solution for the Imagination PowerVR SGX544 GPU on Xen and ARM
  • Xen and OpenStack
  • Evolution of Virtual Machine Introspection (including HW assistance) in the Xen Hypervisor
  • Vendor Strategies For Upgrading Xen in their products (e.g. from Xen 4.1.5 to 4.5)
  • Effectiveness of New Xen Project Security Policy

As usual, we will post summaries (or patches/RFC’s) from these discussions on xen-devel@ – I will also post links to follow-up discussions on our wiki.

Future Xen Project Developer Events in Asia

We’ve learned that the term hackathon is misleading for this event and confuses some of our attendees. Our hackathons are really more of an Architecture Workshop and Design Summit. For this reason, we will probably rename the Hackathon: for a current proposal on the new name check out this and this e-mail thread.

As the event was very successful and we have a growing, active developer community in China, we are considering holding another similar event in 2017 or a Xen Project Developer Summit at LinuxCon Japan in 2017. Stay tuned for more details.

20150427_205419 20150427_205410 20150427_211809 20150429_174059

 

Introducing the Xen Project – OpenStack CI Loop

We recently introduced the new Xen Project Test Lab, a key piece of infrastructure to improve the quality of our code and code coverage. As stated earlier, “we are serious and proactive when it comes to minimising and, whenever possible, eliminating any adverse effects from defects, security vulnerabilities or performance problems”. This also applies to Xen Project integration with OpenStack, which is why the Xen Project Advisory Board made available funds to create the Xen Project – OpenStack CI Loop in January 2015. We started work on setting up our OpenStack CI Loop immediately after the funds were made available, fixed a number of issues in the Xen Project Hypervisor, Libvirt and OpenStack Nova and are excited to announce that our Xen Project – OpenStack CI Loop is now live and in production.

I wanted to thank members of our Test Infrastructure Working Group, comprised of employees from AMD, Amazon Web Services, Citrix, Intel and Oracle, and community members from Rackspace and Suse who have been supporting the creation of the Xen Project – OpenStack CI Loop.

What does an OpenStack CI Loop Do?

An OpenStack external testing platform (or CI Loop) enables third parties to run tests against an OpenStack environment that is configured with that third party’s drivers or hardware and reports the results of those tests on the code review of a proposed OpenStack patch. It is easy to see the benefit of this real-time feedback by taking a look at a code review that shows how these platforms provide feedback.

In this screenshot, you can see a number Verified +1 and one Verified -1 labels added by CI loops to OpenStack Nova.

In this screenshot, you can see a number Verified +1 and one Verified -1 labels added by CI loops to OpenStack Nova.

The figure below, shows the number of OpenStack Nova drivers for Hypervisors, which allow you to choose which Hypervisor(s) to use for your Nova Deployment. Note that these are classified into groups A, B and C.

This diagram shows the different Nova compute drivers and their quality status

This diagram shows the different Nova compute drivers and their quality status

In a nutshell, the groups have the following meaning:

  • Group C: These drivers have minimal testing.
  • Group B: These drivers have unit tests that gate commits and functional testing providing by an external system that does not gate commits, but advises patch authors and reviewers of results in the OpenStack code review system.
  • Group A: These drivers have unit tests that gate commits and functional testing that gate commits.

With the introduction of the Xen Project – OpenStack CI Loop we are close to achieving our first goal of moving the Xen Project Hypervisor from Group C to B. This is a goal we first publicly stated at this year’s FOSDEM’15 where some of you may had had the opportunity to hear Stefano Stabellini talk about using the Xen Project Hypervisor in OpenStack.

What we test against?

Currently, our CI Loop tests all OpenStack Nova commits against Ubuntu Xen 4.4.1 packages with a number of patches applied to them, together Libvirt 1.2.14 with a number of patches applied to them (for more details see this specification). All the patches are already integrated upstream. When off-the shelf packages of these changes are available in Linux distros, we will start testing against them.

What is next?

Monitor the CI loop test results against the official OpenStack CI: we will run the CI loop for a few weeks and monitor, whether there are any discrepancies between the results of the official Nova CI loop and ours. If there are any, we will investigate where the issue is and fix them. This is important, as we had a number of intermittent issues in the Xen Libvirt driver and we want to be sure, we have fixed them all.

Fix two known issues, and enable two disabled tests: We have two test cases related to iSCSI support (test_volume_boot_pattern), which are currently disabled. In one case, we believe the that the Tempest Integration Suite makes some KVM specific assumptions: Xen Project community members are working with the OpenStack community to fix the issue. The second issue is related, but requires further investigation.

Make the Xen Project – OpenStack CI Loop voting: The next logical step, is to work with the OpenStack community to make our CI loop voting. This is the first step to move from group C to B.

Beyond That: Build trust and become an active member of the OpenStack community. Our ultimate goal, is to ensure that the Xen Project Hypervisor moves into group A, alongside KVM. We are also investigating, whether it makes sense to run Xen Project + Libvirt against OpenStack Neutron. We are also looking at integrating OpenStack Tempest into our new Xen Project Test Lab, such that we can ensure that upstream Xen and Libvirt will work with OpenStack Nova.

Can I Help?

If you use Xen Project with OpenStack and Libvirt, feel free to get in touch. Some members of the Xen Project community will be at the Vancouver OpenStack Summit and we are keen to learn, what issues you have and whether we can help fixing them. As community manager, will help arrange meetings and build bridges: feel free to contact me under community dot manager at xenproject dot org. You can also ask questions on xen-devel@ and xen-users@ and connect with Xen Project Developers and Users.

Additional Resources

Introducing Xen Project’s New Test Lab

One of Xen Project’s highest priorities is to continually work to improve the quality of our code and code coverage. We’re serious and proactive when it comes to minimising and, whenever possible, eliminating any adverse effects from defects, security vulnerabilities or performance problems. Our users run some of the largest cloud and datacenter operations in the world, so we know that reboots and service interruptions are more than mere glitches in operations.

That’s why the Xen Project Advisory Board has made a significant investment to set up a Xen Project-owned Test Lab based on OSSTEST. We’re excited to announce that the lab is now live and in production.

Our new test lab will allows us to improve upstream quality of the Xen Project Hypervisor, while developers from different organizations now have access to a system that can be used to investigate test failures, schedule test jobs, etc. The test lab is also being used for regression testing of the Xen Project Hypervisor against different upstream and downstream environments. The Xen Project also tests against upstream and downstream open source projects such as the Linux Kernel and OpenStack to ensure that the Xen Project Hypervisor works seamlessly with important key technologies.

To help kick-start the lab, we created the Test Infrastructure Working Group, which helped source a COLO, spec out the hardware requirements for the test system and helped procure the machines needed to run OSSTEST. A special thank you goes to Ian Jackson, who has been instrumental in getting this effort off the ground over the past six months. We also appreciate contributions from our committee comprised of employees from AMD, Amazon Web Services, Citrix, Intel and Oracle, who have worked closely with our maintainers, project leads and wider community to create the new testing facility. As a result, we’ve decommissioned the old system run by Citrix as a service to the Xen Project community.

In this blog, we’ll describe our platform in more detail, identify what’ we’re doing today with OSSTEST and new goals from our Test Infrastructure Working Group.

Xen Project’s New Test Lab

This picture shows a section of our test lab, which is co-located at Earthlink

This picture shows a section of our test lab, which is co-located at the Earthlink Boston Data Centre

Currently the Xen Project owns a rack with a mixture of 24 x86 Intel and AMD hosts, and 8 ARM hosts (a custom-made set-up that includes 4 Cubietruck and 4 Arndale boards). Although we are currently only at 50 percent of planned capacity, the new test system has a much more diverse set of test machines and already offers more capacity than the old one did. It is also significantly more reliable than the old system.

Setting up a test lab with several different architectures and suppliers as well as a globally dispersed community is simply a challenge. For example, we discovered a number of bugs in the Linux kernel, FreeBSD and Hypervisor, which are currently being addressed. We also tripped over a number of unexpected issues, such as hardware issues and BIOS bugs in some of the machines in the COLO.

How Do We Use OSSTEST Today?

As mentioned earlier, the test lab is being used for regression testing. We’re testing different versions of Xen Project Hypervisor against different versions of the Linux Kernel, Linux distros, FreeBSD, Windows, QEMU, Rump Kernels and other open source projects we interface with. We also do test specific features such as Xen Security Modules, Credit 2 scheduler and other Xen Project features. And more recently we added functionality for performance testing of the Xen Project Hypervisor.

The set-up is essentially following a continuous integration paradigm; changes to Xen Project trees are pushed onto a staging branch. When tests succeed, changes are automatically pushed to the master branch. If there are failures, a bisection algorithm is run, which in many cases will identify the changeset that is causing the issue and developers are then notified that a specific changeset has created a problem.

This figure shows how the interaction between the test lab, the staging branch and the master branch.

This figure shows how the contribution workflow and its interaction between the test lab, the staging branch and the master branch.

What’s Next for Xen Project’s Test Lab

Bring the test lab to 100% capacity. As we work through the remaining issues, new machines will be taken into production as issues are resolved. The increased capacity will enable us to add on-demand testing capability, which will enable community members to test on hardware that they may not have themselves.

Broaden Access: The new system is in a community-operated COLO, rather than one hosted inside the Citrix private network. This, and the increased capacity, will mean that we can to grant access to the facility to Xen Project community members regardless of their affiliation, and provide an on-demand testing capability. To do this, we will need to agree upon processes to enable community members to access the test lab, in a similar way as we allow access to Coverity Scan data.

More Test Coverage: This increased capacity will allow us to further increase the breadth and depth of our testing. We have a number of enhancements in the pipeline, including for example testing of FreeBSD hosts, nested virtualisation, and compatibility testing with a variety of OS distributions.

Expand the infrastructure and add another rack. The Xen Project Advisory Board agreed to fund another rack and to procure further test machines in 2015. We expect that some 64-bit ARM machines will be part of the next tranche of machines that the Xen Project will buy.

Xen Project OpenStack CI loop: In addition to the Xen Project Test Lab, the Advisory Board is also funding a 3rd party continuous integration loop to improve Xen Project virtualization and OpenStack integration. A prototype of the CI loop is already running at jenkins.openstack.xenproject.org. Xen Project community members are currently fixing a number of issues in OpenStack Nova, Libvirt and Xen Project, such that the CI loop can become voting on OpenStack Nova commits.

We also continue to run the Coverity Scan static analysis service, which helps us quickly find, review and resolve Xen Project Hypervisor code defects and vulnerabilities. We’ve done this since 2013 and continue to regularly review and fix reported bugs.

If you’re interested in testing Xen, we encourage developers to join the xen-devel mailing list and ask how you can help. We’re always looking for help extending the set of test cases to tune performance for as wide a set of configurations and features as possible.

Xen Project Participates in Outreachy (formerly OPW)

This is a quick reminder that the Xen Project is again participating in Outreachy (formerly known as OPW/Outreach program for Women).

Outreachy is the successor of the Outreach Program for Women. Please see its page for the information about the May 2015 round of interships.

Outreach Program for Women has been helping women (cis and trans), trans men, and genderqueer people get involved in free and open source software. It provided a supportive community for beginning to contribute any time throughout the year and offered focused internship opportunities twice a year with a number of free software organizations. This work is continued by Outreachy, with the goal of expanding the program to more participants from underrepresented backgrounds.

Round 10 of Outreachy

outreachy-poster-2015-May-AugustThe application deadline for interns is March 24, 2015 April 7, 2015. For a list of projects for interns and more information on how to apply, check our Xen Project Outreachy portal.

We have many different projects in many different areas! Check out the following project ideas: Outreach Program Projects, General List of Development Projects or MirageOS Pioneer Projects.

Learn about the Experience of past Applicants

At least year’s Xen Project Developer Summit, we ran a panel discussion that included OPW interns, GSoC students as well as mentors.


You may also want to read Women interns rocking open source at Xen Project.

GSoC 2015

Unfortunately the project has not been accepted for GSoC in 2015! However, there are several Xen related projects hosted in various mentoring organisations, such as

Please apply to these projects using the normal GSoC application process.