Author Archives: Lars Kurth

About Lars Kurth

Lars Kurth is a highly effective, passionate community manager with strong experience of working with open source communities (Symbian, Symbian DevCo, Eclipse, GNU) and currently is community manager for xen.org. Lars has 9 years of experience building and leading engineering teams and a track record of executing several change programs impacting 1000 users. Lars has 16 years of industry experience in the tools and mobile sector working at ARM, Symbian Ltd, Symbian Foundation and Nokia. Lars has strong analytical, communication, influencing and presentation skills, good knowledge of marketing and product management and extensive background in C/C , Java and software development practices which he learned working as community manager, product manager, chief architect, engineering manager and software developer. If you want to know more, check out uk.linkedin.com/in/larskurth. Personally, Lars has a wide range of interests such as literature, theatre, cinema, cooking and gardening. He is particularly fascinated by orchids and carnivorous plants and has built a rather large collection of plants from all over the world. His love for plants extends into a passion for travel, in particular to see plants grow in their native habitats.

A Recap of the Xen Project Developer Summit 2016

The Xen Project descended on Toronto, Canada in late August for its annual Xen Project Developer Summit. The Summit is an opportunity for developers and software engineers to collaborate and discuss the latest advancements of the Xen Project software. It also gives developers a chance to better understand new trends and deployments in the community and from power enterprise users.

From community growth to new emerging use cases, the Summit covered a lot of ground. Developments within core technologies such as security, graphics support and hardware support were discussed. We also covered emerging technologies such as automotive, embedded and IoT. All sessions were recorded and are available here and also on slideshare (follow this link for summit presentations).

Below is a summary of a few videos that feature technology that has been recently introduced into the Xen Project hypervisor as well as emerging technologies that are being built with Xen Project technology.

New Feature Technologies from Xen Project Community and Power Users

In Xen Project 4.7, we introduced Live Patching as a technology preview. Live Patching gives system administrators and DevOps practitioners the ability to update the Xen Project hypervisor without the need for a reboot. Konrad Wilk, software development manager of Oracle and Ross Lagerwall, software enggineer at Citrix, provide insight into how it works, what the difficulties were to implement, and how it compares to other technologies for patching (kGraft, kPatch, kSplice, Linux hot-patching).



Dimitri Stilliadis, CEO of Aporeto, provides a great overview of the benefits of using Xen Project software to provide an execution environment for Docker apps. This approach allows VM-like isolations for security measures without having to sacrifice performance. The presentation introduces a new paravirtualized protocol to virtualise IP sockets and provides the design and implementation details.


Data breaches are happening all the time, and there are many ways that organisations are trying to stop this through detection, pattern matching and behavioural analysis. However, Neil Sikka, founder and CEO of A1LOGIC, provides a new way of looking at this problem and solving this problem by using the Xen Project hypervisor to enforce data loss prevention. It doesn’t use any type of detection, heuristics, pattern matching or behavioural analysis, but rather a strictly algorithmic approach rooted in hardware.

Embedded Projects and Xen Project Software

Members from the Xen Project sister community OpenXT, an open-source development toolkit for hardware-assisted security research and appliance integration, were present to provide some insights into how Xen Project is working within the embedded space and best practices for embedding Xen Project on mobile and tablet devices.

If this is an area that you are interested in, check out Christopher Clark (consultant and interoperability architect at BAE Systems) overview of the OpenXT Project, which has begun to attract new users and contributors. We also recommend Chris Patterson’s and Kyle Temkin’s step-by-step guide on the challenges and lessons to get Xen Project software started on phones and tablets. Chris is a advising computer engineer for AIS and Kyle is researcher for AIS.


Emerging Technologies

Xen Project is consistently becoming more common within automotive and aviation. Xen Project 4.7 introduced the ability to remove core Xen Hypervisor features at compile time via KCONFIG. This allows a more lightweight hypervisor, which is perfect for IoT scenarios and better for security-first environments, like automotive.

Sangyun Lee, senior embedded software engineer of LG Electronics, presents on the real-time GPU scheduling of XenGT in Automotive Embedded systems. It introduces the real-time GPU schedule of XenGT running on automotive embedded systems and explains why this should be used for an automotive system.


Xen Project is consistently being used within embedded systems for automotive. Earlier this year at CES, GlobalLogic showcased its technology behind Nautilus, which is the company’s virtualisation solution that enables multiple domains to share the GPU hardware with no more than a 5 percent overall in performance changes. More on this technology and how it uses Xen Project here.

The summit was a huge success with many interesting conversations. The Xen Project thanks everyone who attended and presented as well as the sponsors of the event Citrix, Huawei and Intel.

Virtual Machine Introspection: A Security Innovation With New Commercial Applications

The article from Lars Kurth, the Xen Project chairperson, was first published on Linux.com.

A few weeks ago, Citrix and Bitdefender launched XenServer 7 and Bitdefender Hypervisor Introspection, which together compose the first commercial application of the Xen Project Hypervisor’s Virtual Machine Introspection (VMI) infrastructure. In this article, we will cover why this technology is revolutionary and how members of the Xen Project Community and open source projects that were early adopters of VMI (most notably LibVMI and DRAKVUF) collaborated to enable this technology.

Evolving Security Challenges in Virtual Environments

Today, malware executes in the same context and with the same privileges as anti-malware software. This is an increasing problem, too. The Walking Dead analogy I introduced in this Linux.com article is again helpful. Let’s see how traditional anti-malware software fits into the picture and whether our analogy applies to anti-malware software.

In the Walking Dead universe, Walkers have taken over the earth, feasting on the remaining humans. Walkers are active all the time, and attracted by sound, eventually forming a herd that may overrun your defences. They are strong, but are essentially dumb. As we explored in that Linux.com article, people make mistakes, so we can’t always keep Walkers out of our habitat.

For this analogy, let’s equate Walkers with malware. Let’s assume our virtualized host is a village, consisting of individual houses (VMs) while the Hypervisor and network provides the infrastructure (streets, fences, electricity, …) that bind the village together.

Enter the world of anti-malware software: assume the remaining humans have survived for a while and re-developed technology to identify Walkers fast, destroy them quickly and fix any damage caused. This is the equivalent of patrols, CCTV, alarmed doors/windows and other security equipment, troops to fight Walkers once discovered and a clean-up crew to fix any damage. Unfortunately, the reality of traditional malware security technology can only be deployed within individual houses (aka VMs) and not on the streets of our village.

To make matters worse, until recently malware was relatively dumb. However, this has changed dramatically in the last few years. Our Walkers have evolved into Wayward Pine’s Abbies, which are faster, stronger and more intelligent than Walkers. In other words, malware is now capable of evading or disabling our security mechanisms.

What we need is the equivalent of satellite surveillance to observe the entire village, and laser beams to remotely destroy attackers when they try and enter our houses. We can of course also use this newfound capability to quickly deploy ground troops and clean-up personnel as needed. In essence that is the promise that Virtual Machine Introspection gives us. It allows us to address security issues from outside the guest OS without relying on functionality that can be rendered unreliable from the ground. More on that topic later.

From VMI in Xen to the First Commercial Application: A Tale of Collaboration

The development of Virtual Machine Introspection and its applications show how the Xen Project community is bringing revolutionary technologies to market.

The development of Virtual Machine Introspection and its applications show how the Xen Project community is bringing revolutionary technologies to market.

The idea of Virtual Machine Introspection for the Xen Project Hypervisor hatched at Georgia Tech in 2007, building on research by Tal Garfinkel and Mendel Rosenblum in 2003. The technology was first incorporated into the Xen Project Hypervisor via the XenAccess and mem-events APIs in 2009. To some degree, this was a response to VMware’s VMsafe technology, which was introduced in 2008 and deprecated in 2012, as the technology had significant limitations at scale. VMSafe was replaced by vShield, which is an agent-based, hypervisor-facilitated, file-system anti-virus solution that is effectively a subset of VMsafe.

Within the Xen Project software however, Virtual Machine Introspection technology lived on due to strong research interests and specialist security applications where trading off performance against security was acceptable. This eventually led to the creation of LibVMI (2010), which made these APIs more accessible. This provided an abstraction that eventually allowed exposure of a subset of Xen’s VMI functionality to other open source virtualization technologies such as KVM and QEMU.

In May 2013, Intel launched its Haswell generation of CPUs, which is capable of maintaining up to 512 EPT pointers from the VMCS via the #VE and VMFUNC extensions. This proved to be a potential game-changer for VMI, enabling hypervisor controlled and hardware enforced strong isolation between VMs with lower than previous overheads, which led to a collaboration of security researchers and developers from Bitdefender, Cisco, Intel, Novetta, TU Munich and Zentific. From 2014 to 2015, the XenAccess and mem-events APIs have been re-architected into the Xen Project Hypervisor’s new VMI subsystem, alt2pm and other hardware capabilities have been added, as well as support for ARM CPUs and a baseline that was production ready has been released in Xen 4.6.

Citrix and Bitdefender collaborated to bring VMI technology to market: XenServer 7.0 introduced its Direct Inspect APIs built on the Xen Projects VMI interface. It securely exposes the introspection capabilities to security appliances, as implemented by Bitdefender HVI.

What Can Actually Be Done Today?

Coming back to our analogy: what we need is the equivalent of satellite surveillance to observe the entire village. Does VMI deliver? In theory, yes: VMI makes it possible to observe the state of any virtual machine (house and its surroundings in the village), including memory and CPU state and to receive events when the state of the virtual machine changes (aka if there is any movement). In practice, the performance overhead of doing this is far too high, despite using hardware capabilities.

In our imagined world that is overrun by Walkers and Abbies, this is equivalent to not having the manpower to monitor everything, which means we have to use our resources to focus on high value areas. In other words, we need to focus on the suspicious activity on system perimeters (the immediate area surrounding each of our houses).

This focus is executed by monitoring sensitive memory areas for suspicious activity. When malicious activity is detected, a solution can take corrective actions on the process state (block, kill) or VM state (pause, shutdown) while collecting and reporting forensic details directly from a running VM.

Think of a laser beam on our satellite that is activated whenever an Abbie or Walker approaches our house. In technical terms, the satellite and laser infrastructure maps to XenServer’s Direct Inspect API, while the software which controls and monitors our data maps onto Bitdefenders Hypervisor Introspection.

It is important to stress that monitoring and remedial action takes place from the outside, using the hypervisor to provide hardware-enforced isolation. This means that our attackers cannot disable surveillance nor laser beams.

Of course, no security solution is perfect. This monitoring software may not always detect all suspicious activity, if that activity does not impact VM memory. This does not diminish the role of file-system-based security; we must still be vigilant, and there is no perfect defense. In our village analogy, we could also be attacked through underground infrastructure such as tunnels and canalisation. In essence this means we have to use VMI together with traditional anti-malware software.

How does VMI compare to traditional hypervisor-facilitated anti-virus solutions such as vShield? In our analogy, these solutions require central management of all surveillance equipment that is installed in our houses (CCTV, alarmed doors/windows, …) while the monitoring of events is centralized very much like a security control centre in our village hall. Albeit such an approach significantly simplifies monitoring and managing of what goes on within virtual machines, it does not deliver the extra protection that introspection provides.

You can find more information (including some demos) about VMI, XenServer Direct Inspect API and BitDefender Hypervisor Introspection here:

Xen Project Virtual Machine Introspection

Conclusion

The development of VMI and its first open source and commercial applications show how the Xen Project community is innovating in novel ways, and is capable of bringing revolutionary technologies to market. The freedom to see the code, to learn from it, to ask questions and offer improvements has enabled security researchers and vendors such as Citrix and Bitdefender to bring new solutions to market.

It is also worth pointing out that hardware-enabled security technology is moving very fast: only a subset of Intel’s #VE and VMFUNC extensions are currently being deployed in VMI. Making use of more hardware extensions carries the promise of combining the protection of out-of-guest tools with the performance of in-guest tools.

What is even more encouraging is that other vendors such as A1Logic, Star Lab and Zentific are working on new Xen Project-based security solutions. In addition, the security focused, Xen-based OpenXT project has started to work more closely with the Xen Project community, which promises further security innovation.

A few of these topics will be discussed in more detail during Xen Project Developer Summit happening in Toronto, CA from August 25 – 26, 2016. You learn more about the event here.

Q&A: Xen Project Release Strengthens Security and Pushes New Use Cases

The following Q&A with Lars Kurth, the Xen Project chairperson, was first published on Linux.com.

Xen Project technology supports more than 10 million users and is a staple in some of the largest clouds in production today, including Amazon Web Service, Tencent, and Alibaba’s Aliyun. Recently, the project announced the arrival of Xen Project Hypervisor 4.7. This new release focuses on improving code quality, security hardening and features, and support for the latest hardware. It is also the first release of the project’s fixed-term June – December release cycles. The fixed-term release cycles provide more predictability making it easier for consumers of Xen to plan ahead.

We recently sat down with the Xen Project chairperson, Lars Kurth, to talk about some of the key features of the release and the future of Xen Project technology. Lars will be discussing this topic and more during Xen Project’s Developer Summit in Toronto, CA from August 25-26 — the conference is directly after LinuxCon North America.

Q: What was the focus on this release?

Lars Kurth: There were five areas that we focused on for this release (full details are in our blog). In summary, we focused on security features, migration support, performance and workloads, support for new hardware features, and drivers and devices (Linux, FreeBSD and other).

Security is consistently something that we focus on in all of our releases. There are a lot of people that rely on Xen Project technology and security is our top concern in any release as well as how we organize our process around security disclosures.

Q: What was the biggest feature coming out of this release?

Lars: The biggest feature for us is live patching, which is a technology that enables re-boot free deployment for security patches to minimize disruption and downtime during security upgrades for cloud admins. It essentially eliminates all cloud reboots, making cloud providers and their users much more safe. It also eliminates a lot of headaches for system and DevOps admins of the world.

Q: Xen is often associated with the cloud, but are there additional use cases that you see growing around this technology, if so why?

Lars: We are seeing a lot of growth in terms of contributions, as well as many different use cases emerging, including automotive, aviation, embedded scenarios, security, and also IoT. In addition, we continue to grow within the public cloud sector and traditional server virtualization.

On the security front, for example, a number of vendors such as A1Logic, Bitdefender, Star Lab and Zentific have released or are working on new Xen Project-based security solutions. In addition, the security focused and Xen-based OpenXT project has started to work more closely with the Xen Project community.

Long-time contributors to the Xen Project, such as DornerWorks – a premier provider of electronic engineering services for the aerospace, medical, automotive, and industrial markets – have expanded their scope and are now providing support for the Xen Xilinx Zynq Distribution targeting embedded use-cases. We have also seen an increasing number of POCs and demos of automotive solutions, which include Xen as a virtualization solution.

Growth in these sectors is largely due to the Xen Project’s flexibility, extensibility, customisability and a clear lead when it comes to security-related technologies. Over the last year, we have also seen contributions increase from developers with strong security and embedded backgrounds. In fact, this totaled nearly 17 percent of the overall contributions in this release cycle, up from 9 percent in the previous release.

Q: How did you address these uses cases in this latest release?

Lars: We introduced the ability to remove core Xen Project Hypervisor features at compile via KCONFIG. This creates a more lightweight hypervisor and eliminates extra attack surfaces that are beneficial in security-first environments and microservice architectures. Users will still be able to get the core hypervisor functions, but they won’t receive all the drivers, schedulers, components or features that might not fit their use case.

Essentially it gives people an “a la carte” feature set. They can decide what they need for compliance, safety or performance reasons.

Q: Were there any new contributors for this release that surprised you?

Lars: We had three new companies contributing to the project: Star Lab, Bosch and Netflix. I met engineers from Star Lab for the first time at the 2015 Developer Summit less than a year ago, and helped introduce them to the Project’s culture. In that short period of time, Doug Goldstein from Star Lab has moved into the top five contributors and top 10 code reviewers for the Project.

I was surprised about Netflix’s contributions; I didn’t even know the company used Xen. Netflix improved and secured the VPMU feature, which is incredibly useful for system tuning and performance monitoring. Bosch Car Multimedia GmbH added some new ARM functionality. In addition, we have seen quite a bit of Xen related development in upstream and downstream projects such as Linux, FreeBSD, NetBSD, OpenBSD, QEMU and Libvirt.

Q: What’s next for Xen Project? Where do you think the technology is heading in the future and why?

Lars: In the last three releases, we introduced several major new features such as PVH, COLO, new schedulers, VMI, Live Patching, Graphics Virtualization, etc. and significant re-work of existing features such as Migration and the Xen Security Modules (XSM). Looking at trends within the community, I expect that stepwise evolution of large new features to continue.

Some new capabilities, such as restartable Dom0’s, and additional techniques to provide more isolation and security, are also likely to appear. In addition, it looks likely that we will see some GPU virtualization capabilities for GPUs that target the ARM ecosystem, although it is not yet clear whether these will be available as open source. I also expect that both Intel and ARM hardware features will be closely tracked.

Some areas, such as new schedulers, XSM, PVH and Live Patching, will see significant efforts to harden and improve existing functionality. The goal is to ensure their swift adoption in commercial products and Linux and BSD distributions. Some features, which are not enabled by default are likely to become part of the Xen Project Hypervisor’s default configuration.

Xen Project 4.7 and 4.6.3 Release

I’m pleased to announce the release of Xen Project Hypervisor 4.7 and Xen Project Hypervisor 4.6.3.

Xen Project Hypervisor 4.7

This new release focuses on improving code quality, security hardening, security features, live migration support, usability improvements and support for new hardware features — this is also the first release of our fixed term June – December release cycle.

We continue to strive to make Xen Project Hypervisor the most secure open source hypervisor to match the security challenges in cloud computing, and for embedded and IoT use-cases. We are continuing to improve upon the performance and scalability for our users, and aim to continuously bring many new features to our users in a timely manner.

To make it easier to understand the major changes during this release cycle, I’ve grouped them below into several categories:

  • Security Features
  • Migration Support
  • Performance and Workloads
  • Support for new Hardware Features
  • Drivers and Devices (Linux, FreeBSD and other)

Security Features

Reboot-free Live Patching: Xen Project Hypervisor 4.7 comes equipped with Live Patching, a technology that enables re-boot free deployment of security patches to minimize disruption and downtime during security upgrades for system administrators and DevOps practitioners. Xen Project 4.7 implements version 1 of the Xen Project’s Live Patching specification, which is designed to encode the vast majority of security patches (approximately 90%) as Live Patching payloads. This version ships with a Live Patching enabled hypervisor and payload deployment tools and is available as a technology preview.

KCONFIG support: For security, embedded automotive and IoT use cases, Xen Project introduced the ability to remove core Xen Hypervisor features at compile time via KCONFIG. This ability creates a more lightweight hypervisor and eliminates extra attack surfaces that are beneficial in security-first environments, microservice architectures and environments that have heavy compliance and certification needs, like automotive.

Improvements to the Virtual Machine Introspection (VMI) subsystem: A number of performance, scalability, robustness and interface improvements have been added to the Virtual Machine Introspection subsystem, that was introduced in Xen 4.5. In addition, Bitdefender Hypervisor Introspection leveraging Xen Project Virtual Machine Introspection, has recently been released as a new enterprise security solution to discover and remedy deep threats that remain hidden via traditional endpoint security tools.

Foundation work to tolerate a restartable Dom0: Several key components in a Xen Project system run in Dom0, which make Dom0 the single point of failure. Xen Project has been able to run xenstored, the daemon for managing the hypervisor’s central settings repository on a Xen Project host, in a sandboxed Virtual Machine called xenstored stub domain since Xen Project version 4.2. In Xen 4.7, we have made it easier to build xenstored stub domains and for them to tolerate a Dom0 restart. This will make Dom0 less critical to a Xen Project system and help us move towards a more robust and secure architecture in the future. More work in this area is expected in subsequent releases.

Migration Support

Improved Migration support: CPU ID Levelling enables migration of VM’s between a larger range of non-identical hosts than previously supported.

Fault Tolerance / Coarse-grained Lock-stepping (COLO): Xen 4.5 laid the foundation for COLO while improving the Xen Project’s Hypervisors Live Migration and Remus High Availability support. The COLO Manager, which introduces a relaxed approach to checkpointing that avoids unnecessary checkpoints enabling near native performance for many workloads, has been fully integrated as an experimental feature into Xen 4.7. Note that the COLO Block Replication and COLO Proxy components, both of which are QEMU components, are currently still reviewed by the QEMU community. Both components are available as out-of-tree add-ons to the Xen Project Hypervisor, until fully integrated into QEMU.

Performance and Workloads

Support for a wider range of workloads and applications: The PV guest limit restriction of 512GB has been removed to allow the creation of huge PV domains in the TB range. TB sized VMs, coupled with Xen Project’s existing support for 512 vCPUs per VM, enable execution of memory and compute intensive workloads such as big data analytics workloads and in-memory databases.

Improved Credit 2 scheduler: The Credit2 scheduler is one (big) step closer to being ready for production use. It is now possible to instruct the scheduler to organize its runqueues and perform load balancing at core, socket or NUMA node granularity. More fine grained (core) configurations, deliver more aggressive load balancing, and are best suited for medium size systems. This feature has been proven to enable very good performance, especially if Hyper Threading is present.

Less fine grained configurations entail less overhead, and is suitable for larger servers or when no Hyper Threading is available. In addition, Credit2 has been extended to allow pinning of vCPUs to pCPUs (also known as “hard affinity”), allowing system administrators to configure the system in the exact way they want, and achieve the best setup for a given workload (for instance, a guarantee that a certain subset of vCPUs are always able to run when they need to run).

Improved RTDS scheduler: The RTDS scheduler is a real-time CPU scheduler built to provide guaranteed CPU capacity to guest VMs on SMP hosts, which primarily targets embedded, real-time and low-latency workloads. In Xen Project 4.7, the scheduling model has been changed from a quantum-driven to an event-driven model, which reduces scheduling overhead and thus scalability and performance for embedded and realtime workloads. In addition, per-VCPU parameter configuration has been added to allow better scheduler control for specialised workloads.

Per-cpu reader-writer lock: This new infrastructure allows for the fast path read case to have low overhead by only setting/clearing a per-cpu variable for using the read lock. After transforming various hypervisor locks to this infrastructure, VM-VM network transfer with 16 queues jumped from 15 gbit/s to 48 gbit/s on a 2 socket Haswell-EP host.

Usability Improvements

PVUSB Support: In Xen Project 4.7, a new XL command line interface to manage PVUSB devices has been introduced to manage PVUSB devices for PV guests. Both in kernel PVUSB backend and QEMU backend are supported.

Hot plugging of QEMU disk backends: Xen Project now enables hot-plugging of USB devices as well as QEMU disk backends, such as drbd, iscsi, and more in HVM guests. This new feature allows users to add and remove disk backends to virtual machines without the need to reboot the guest.

Soft-reset: The soft reset feature for HVM guests allows for a more graceful shutdown and restart of the HVM guest.

New Hardware Support

Features specific to the ARM Architecture

SBBR Compliance: Xen Project now supports booting on hosts that expose ACPI 6.0 (and later) information. The ARM Server Base Boot Requirements (SBBR) stipulate that compliant systems need to express hardware resources with ACPI; thus this support will come in useful for ARM Servers. This effort was carried out by Shannon Zhao of Linaro with minor patches from Julien Grall of ARM.

PCSI 1.0 Compatibility: PSCI 1.0 compatibility allows Xen Project software to operate on systems that expose PSCI 1.0 methods. Now, all 1.x versions of PSCI will be compatible with Xen Project software. More information on Power State Coordination Interface can be found here. This effort was also carried out by Julien Grall with a patch from Dirk Behme of Bosch.

vGIC-v3: Virtual Generic Interrupt Controller version 3. Reworked to be spec-compliant and optimised in some code paths.

Wallclock support: ARM guest can now get wallclock time directly from Xen Project via shared info page.

Features specific to Intel® Xeon® processor product family

Improved Interrupt Efficiency: Xen Project 4.7 supports VT-d Posted Interrupts, which provides hardware-level acceleration to increase interrupt virtualization efficiency. It reduces latency and improves user experience through performance improvements, especially for interrupt-intensive front-end workloads such as web servers. Note that Posted Interrupts in Xen Project 4.7 are still experimental and disabled by default.

Code and Data Prioritization: Xen Project 4.7 is the first to include Code and Data Prioritization (CDP), part of the Intel® Resource Director Technology (RDT) Framework and an extension of Cache Allocation Technology (CAT), first introduced in Xen Project 4.6. The introduction of CDP allows isolation of code/data within the shared L3 cache of multi-tenant environments, reducing contention and improving performance.

Other Intel Features: Additional features specific to the Intel Xeon processor family in Xen Project 4.7 include: VMX TSC Scaling, which allows for easier migration between machines with different CPU frequencies and support for Memory Protection Keys, a new security feature for hardening the software stack.

Drivers and Devices (Linux, FreeBSD and other)

During the Xen Project 4.7 release cycle, we made significant improvements to major operating systems and components we rely on to improve interoperability. During this development cycle 1494 Xen Project only related changesets – mostly bug fixes and small improvements – were applied to Linux, FreeBSD, NetBSD, QEMU and the Windows PV drivers: more than twice as many as in the 4.6 release cycle.

Summary

With dozens of major improvements, many more bug fixes and small improvements, and significant improvements to Drivers and Devices, Xen Project 4.7 reflects a thriving community around the Xen Project Hypervisor.

We are extremely proud of achieving the highest quality of the release while increasing development velocity across the hypervisor and its upstream dependencies by about 16%. In particular, our latest security related features enable Xen Project software to compete in the security appliance market and help answer some of the difficult questions regarding security in the cloud era.

We set out at the beginning of this release cycle to foster greater collaboration among vendors, individual developers, upstream maintainers, other projects and distributions. During this release cycle we continued to see an increasing influx of patches and newcomers such as Star Lab, Bosch and Netflix. We had a significant amount of contributions from cloud providers, software vendors, hardware vendors, academic researchers and individuals to help with this release. Major contributors for this particular release come from Citrix, SUSE, Intel, Star Lab, Oracle, Linaro, Fujitsu, Bitdefender, Red Hat, Huawei, ARM, Novetta, Broadcom, Xilinx, Bosch, AMD, GlobalLogic, NSA, Netflix and a number of universities and individuals. Thank you to all who participated.

As the release manager, I would like to thank everyone for their contributions (either in the form of patches, bug reports or packaging efforts) to the Xen Project. This release wouldn’t have happened without contributions from so many people around the world. Please check out our 4.7 contributor acknowledgement page.

The source can be located in the http://xenbits.xen.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.7 tree (tag RELEASE-4.7.0) or can be downloaded as tarball from our website. More information can be found at

Xen Project Hypervisor 4.6.3

The Xen Project 4.6.3 release is a maintenance release which comprises bug fixes and security updates. This is release is available immediately from its git repository
http://xenbits.xen.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.6
(tag RELEASE-4.6.3) or from the Xen Project download page
http://www.xenproject.org/downloads/xen-archives/xen-46-series/xen-463.html
(where a list of changes can also be found).

We recommend all users of the 4.6 stable series which do not wish to upgrade to Xen 4.7, to update to this latest point release.

Note regarding version numbering: an issue was found late in the release process,
after one of the affected qemu trees was already tagged with a signed release git tag. Signed git tags provide a secure way of accounting for the source code, but once created they cannot be removed. Thus, the project could have released this maintenance release with a known issue, or fix the issue and skip a version number. We opted for the latter and decided to skip version 4.6.2.

Xen Project Community Hosts Annual Developers Summit in August

We recently announced the program and speakers for our Xen Project Developer Summit happening in Toronto, Canada from August 25-26, 2016. The event will be co-located with LinuxCon North America.

The Xen Project hypervisor powers the new needs of computing and virtualization through a rich ecosystem of community members that focus on everything from security, embedded, and web-scale environments. The Summit is an opportunity for developers and software engineers to collaborate and discuss the latest advancements of Xen Project software, and better understand what’s next for Xen Project technology, virtualization and cloud computing.

In addition to presentations, we will be running a half-day hackathon alongside the Summit on the last day. Xen Project hackathons have evolved in format into a series of structured problem-solving sessions that scale up to 50 people.

This flagship event features presentations on the latest developments, best practices, collaboration, product roadmap updates and future planning from developers and users who are leading the way in server density, hardware, automotive, cloud and enterprise security. To view the full schedule and register, please head here: http://events.linuxfoundation.org/events/xen-project-developer-summit/program/schedule

This event is being sponsored by Citrix (Diamond sponsor), Huawei (Platinum sponsor) and Intel (Platinum sponsor). Please be sure to follow updates on the event via Xen Project’s Twitter, Google+ or Facebook page. Hashtag for the event is #xendevsummit.

We look forward to seeing you there!

Introducing the Xen Project Code Review Dashboard

The Xen Project’s code contributions have grown more than 10% each year. Although growth is extremely healthy to the project as a whole, it has its growing pains. For the Xen Project, it led to issues with its code review process: maintainers believed that their review workload increased and a number of vendors claimed that it took significantly longer for contributions to be upstreamed, compared to the past.

The project developed some basic scripts that correlated development list traffic with git commits, which showed indeed that it took longer for patches to be committed. In order to identify possible root causes, the project initially ran a number of surveys to identify possible causes for the slow down. Unfortunately, many of the observations made by community members contradicted each other, and were thus not actionable. To solve this problem, the Xen Project worked with Bitergia, a company that focuses on analyzing community software development processes, to better understand and address the issues at hand. We worked with Bitergia on an initial statistical analysis of the code review process and later on a Code Review Dashboard for use by the community. The following OSCON presentation lays out the journey, the project went through:

Findings of the Initial Code Review Study

There were three key areas that we found that were causing the slow down:

  • Huge growth in comment activity from 2013 to 2015
  • We also saw that the time it took to merge patches (=time to merge) increased significantly from 2012 to the first half of 2014. However, from the second half of 2014 time to merge moved back to its long term average. This was a strong indicator that the pre-emptive measures we took, such as a focus on design and architecture reviews and contributor training, actually had an effect.
  • Looking at this in more detail, it turned out that complex patches were taking significantly longer to merge than small patches. As it turns out, a significant number of new features were actually rather complex. At the same time, the demands on the project to deliver better quality and security had also raised the bar for what could be accepted, which impacted new contributors more than established ones.

Introducing the Xen Project Code Review Dashboard

To make the tooling that we developed more accessible to the entire community, the Xen Project Advisory Board funded the development of a Code Review Dashboard. We defined a set of use-cases and supporting data that broadly covered three areas:

  • Community use cases to encourage desired behaviour: this would be metrics such as real review contributions (not justed ACKed-by and Reviewed-by flags), comparing review activity against contributions.
  • Performance use cases that would allow us to spot issues early: these would allow us to filter time related metrics by a number of different criteria such as complexity of a patch series.
  • Backlog use cases to optimize process and focus. The intention here was to give contributors and maintainers tools to see what reviews are active, nearly complete, complete or stale.

To find out more check out

Value for other projects

Like many FOSS projects, the Xen Project code review process uses a mailing list-based review process, and this could be a good blueprint for projects that are finding themselves in the same predicament. It is already clear, that there are many useful additions that can in future be added to the technology we have developed. In addition, we are currently working on improvements of the Code Review Dashboard as part of an Outreachy Project by Priya V (check out her blog), which is jointly mentored by Lars Kurth (Xen Project) and Jesus M. Gonzalez-Barahona (Bitergia). All the code is open source: if you want to have a look, check out the code and the contribution guide.