Tag Archives: code review dashboard

Introducing the Xen Project Code Review Dashboard

The Xen Project’s code contributions have grown more than 10% each year. Although growth is extremely healthy to the project as a whole, it has its growing pains. For the Xen Project, it led to issues with its code review process: maintainers believed that their review workload increased and a number of vendors claimed that it took significantly longer for contributions to be upstreamed, compared to the past.

The project developed some basic scripts that correlated development list traffic with git commits, which showed indeed that it took longer for patches to be committed. In order to identify possible root causes, the project initially ran a number of surveys to identify possible causes for the slow down. Unfortunately, many of the observations made by community members contradicted each other, and were thus not actionable. To solve this problem, the Xen Project worked with Bitergia, a company that focuses on analyzing community software development processes, to better understand and address the issues at hand. We worked with Bitergia on an initial statistical analysis of the code review process and later on a Code Review Dashboard for use by the community. The following OSCON presentation lays out the journey, the project went through:

Findings of the Initial Code Review Study

There were three key areas that we found that were causing the slow down:

  • Huge growth in comment activity from 2013 to 2015
  • We also saw that the time it took to merge patches (=time to merge) increased significantly from 2012 to the first half of 2014. However, from the second half of 2014 time to merge moved back to its long term average. This was a strong indicator that the pre-emptive measures we took, such as a focus on design and architecture reviews and contributor training, actually had an effect.
  • Looking at this in more detail, it turned out that complex patches were taking significantly longer to merge than small patches. As it turns out, a significant number of new features were actually rather complex. At the same time, the demands on the project to deliver better quality and security had also raised the bar for what could be accepted, which impacted new contributors more than established ones.

Introducing the Xen Project Code Review Dashboard

To make the tooling that we developed more accessible to the entire community, the Xen Project Advisory Board funded the development of a Code Review Dashboard. We defined a set of use-cases and supporting data that broadly covered three areas:

  • Community use cases to encourage desired behaviour: this would be metrics such as real review contributions (not justed ACKed-by and Reviewed-by flags), comparing review activity against contributions.
  • Performance use cases that would allow us to spot issues early: these would allow us to filter time related metrics by a number of different criteria such as complexity of a patch series.
  • Backlog use cases to optimize process and focus. The intention here was to give contributors and maintainers tools to see what reviews are active, nearly complete, complete or stale.

To find out more check out

Value for other projects

Like many FOSS projects, the Xen Project code review process uses a mailing list-based review process, and this could be a good blueprint for projects that are finding themselves in the same predicament. It is already clear, that there are many useful additions that can in future be added to the technology we have developed. In addition, we are currently working on improvements of the Code Review Dashboard as part of an Outreachy Project by Priya V (check out her blog), which is jointly mentored by Lars Kurth (Xen Project) and Jesus M. Gonzalez-Barahona (Bitergia). All the code is open source: if you want to have a look, check out the code and the contribution guide.

Disclosure process poll results

Monday we closed the poll for the security discussion. Thank you everyone who participated! The process has not turned up a hidden option that everyone agreed on; however, it has helped find what I hope will be a “median” option which best addresses the concerns and desires as the community as a whole. Below I give a description of the results of the poll, and a recommendation as to what I think is the best way forward.
Continue reading

The Intel SYSRET privilege escalation

The Xen Security team recently disclosed a vulnerability, Xen Security Advisory 7 (CVE-2012-0217), which would allow guest administrators to escalate to hypervisor-level privileges. The impact is much wider than Xen; many other operating systems seem to have the same vulnerability, including NetBSD, FreeBSD, some versions of Microsoft Windows (including Windows 7).

So what was the vulnerability? It has to do with a subtle difference in the way in which Intel processors implement error handling in their version of AMD’s SYSRET instruction. The SYSRET instruction is part of the x86-64 standard defined by AMD. If an operating system is written according to AMD’s spec, but run on Intel hardware, the difference in implementation can be exploited by an attacker to write to arbitrary addresses in the operating system’s memory. This blog will explore the technical details of the vulnerability.
Continue reading

Xen @ Build a Cloud Day Boston

As I mentioned in the Xen Day post, Xen.org was offered a slot at the Build an Open Source Cloud Day Boston. The Build a Cloud attendees were great. They were very engaged and asked lots of questions. The questions gave me a chance to cover several Xen topics that I tend to take for granted. In this post I’ll outline the talk that I gave and highlight some of the questions that were asked and discuss the answers.

The slides for the talk are available here: http://www.slideshare.net/deshantm/why-choose-xen-for-your-cloud

I decided to name the talk “Why Choose Xen For Your Cloud?” so that I could cover the history of Xen in the cloud, specific architectural advantages of Xen, and then cover XCP and Project Kronos. I don’t think that a lot of people realize that “Global Public Computing”  (or Cloud Computing as we know it today) was an idea that led to Xen. In other words, the Xen hypervisor was the solution to the need to break up, isolate, and provide accounting for system resources. Xen was born in the cloud. It’s no surprise then that Amazon EC2, Slicehost (now Rackspace Cloud Servers), and many others have chosen Xen as the basis of their clouds.

As I mentioned about Xen Dom0 support getting into the mainline Linux kernel, I got my first question (something like) “Isn’t Xen support in the Linux kernel going away due to KVM?”. To which I answered no, but let me clarify.  The question makes more sense for someone who is both a Linux expert and virtualization novice since the Linux kernel community is very strict about what it allows into the kernel. For instance, multiple implementations of the same functionality will not be accepted, but an abstracted version that can support more than one specific implementation is often required (OpenVZ is not in mainline, but Linux container components, often inspired by OpenVZ concepts, are included) . The main reason that Xen Dom0 support and KVM are both in the Linux kernel is that Xen and KVM have different architectures. Xen is a type-!, stand-alone hypervisor and KVM is a type-2, integrated hypervisor (I dig into these concepts again soon). Next to further clarify, Xen the hypervisor will never be included in Linux nor is it intended to be. When Xen was first created the Linux kernel at the time was used as a basis, but in a very special way (as explained by Keir Fraser) – the kernel was cut horizontally and the low level bits became the Xen hypervisor and the top part became the first Dom0. The Xen hypervisor is first to the hardware, followed by guest domains (including Dom0), by intentional design and is a fundamental requirement for being a type-1 hypervisor.

Next, as I covered the basic architectural design of a Xen system and its security benefits I was asked to clarify the difference between a type-1 and type-2 hypervisor. I find that the best way to explain this in a practical way that others can quickly understand is by using VMware ESX (the bare-metal, type-1 hypervisor from VMware) and VMware Workstation (the workstation class, install on top of an existing OS, type-2 hypervisor from VMware) since VMware is very well known and people can understand the difference. This question and answer are a perfect lead-in for the architectural and security advantages of Xen. In short, Xen has a small, well-defined, clean, and disaggregatable trusted computing base. This is in contrast to type-2 solutions, but also in contrast to type-1 hypervisors that put more services (perhaps than necessary or desired) into the hypervisor itself. Xen has unique security advantages primarily due to its well-architected design.

Since we were given an hour and fifteen minute slot, the XCP slides are fleshed out a bit more than I have presented in the past. Fortunately, our Xen Day team had prepared some really great materials and I was able to easily leverage those materials and re-focus them for the Build A Cloud/USENIX LISA audience. In particular, it is important to note the XAPI class diagram and cover the basic structure in some detail. The idea being that it helps system admins become familiar with the terminology they will need to master the xe command, but also that these devops-minded admins will likely be interested in writing scripts/tools that make direct XAPI calls. XAPI is great API for the cloud.

The rest of the XCP discussion is centered around why XCP is a great cloud platform. One of the key concepts that XCP, Open vSwitch, and the management tools address is mult-tenancy (multiple independent entities that need to share the same cloud and yet be isolated). XCP really shines for this cloud-minded audience and at least a few people were convinced to skip out on some of the Build a Cloud Day sessions and attend some of the Xen Day afternoon sessions to find out more about XCP. Xen Day took a deep dive into XCP.

XCP and cloud orchestration is an incredible story especially considering that XCP 1.0 was released in 2011. XCP is used in the first commercially available OpenStack cloud from Internap. CloudStack, which is a mature and feature-rich product itself, has also added support for XCP. OpenNebula also recently added support for XCP.  XCP is a great de-facto open source platform to build a cloud on.

I’ve spent a lot of time (and presented in more detail) presentations on Project Kronos, which in a very short time has been developed and is nearing a initial release. Stay tuned to the blog for updates on Project Kronos status. Current documentation on installing and using Project Kronos is available on the Xen wiki:


Xen ARM Project – Did you know about this?

The Xen.org community is currently working on several projects that don’t receive much attention but are critical to the overall success of the Xen hypervisor solution. For example, the Xen ARM project being led by Samsung continues to develop a Xen hypervisor solution for the ARM 9 processor family. Samsung has demonstrated this solution at the past few Xen Summit events and continues to ensure the project’s movement forward.

For more information on the Xen ARM project, go to the following: