A Recap of the Xen Project Developer Summit 2016

The Xen Project descended on Toronto, Canada in late August for its annual Xen Project Developer Summit. The Summit is an opportunity for developers and software engineers to collaborate and discuss the latest advancements of the Xen Project software. It also gives developers a chance to better understand new trends and deployments in the community and from power enterprise users.

From community growth to new emerging use cases, the Summit covered a lot of ground. Developments within core technologies such as security, graphics support and hardware support were discussed. We also covered emerging technologies such as automotive, embedded and IoT. All sessions were recorded and are available here and also on slideshare (follow this link for summit presentations).

Below is a summary of a few videos that feature technology that has been recently introduced into the Xen Project hypervisor as well as emerging technologies that are being built with Xen Project technology.

New Feature Technologies from Xen Project Community and Power Users

In Xen Project 4.7, we introduced Live Patching as a technology preview. Live Patching gives system administrators and DevOps practitioners the ability to update the Xen Project hypervisor without the need for a reboot. Konrad Wilk, software development manager of Oracle and Ross Lagerwall, software enggineer at Citrix, provide insight into how it works, what the difficulties were to implement, and how it compares to other technologies for patching (kGraft, kPatch, kSplice, Linux hot-patching).



Dimitri Stilliadis, CEO of Aporeto, provides a great overview of the benefits of using Xen Project software to provide an execution environment for Docker apps. This approach allows VM-like isolations for security measures without having to sacrifice performance. The presentation introduces a new paravirtualized protocol to virtualise IP sockets and provides the design and implementation details.


Data breaches are happening all the time, and there are many ways that organisations are trying to stop this through detection, pattern matching and behavioural analysis. However, Neil Sikka, founder and CEO of A1LOGIC, provides a new way of looking at this problem and solving this problem by using the Xen Project hypervisor to enforce data loss prevention. It doesn’t use any type of detection, heuristics, pattern matching or behavioural analysis, but rather a strictly algorithmic approach rooted in hardware.

Embedded Projects and Xen Project Software

Members from the Xen Project sister community OpenXT, an open-source development toolkit for hardware-assisted security research and appliance integration, were present to provide some insights into how Xen Project is working within the embedded space and best practices for embedding Xen Project on mobile and tablet devices.

If this is an area that you are interested in, check out Christopher Clark (consultant and interoperability architect at BAE Systems) overview of the OpenXT Project, which has begun to attract new users and contributors. We also recommend Chris Patterson’s and Kyle Temkin’s step-by-step guide on the challenges and lessons to get Xen Project software started on phones and tablets. Chris is a advising computer engineer for AIS and Kyle is researcher for AIS.


Emerging Technologies

Xen Project is consistently becoming more common within automotive and aviation. Xen Project 4.7 introduced the ability to remove core Xen Hypervisor features at compile time via KCONFIG. This allows a more lightweight hypervisor, which is perfect for IoT scenarios and better for security-first environments, like automotive.

Sangyun Lee, senior embedded software engineer of LG Electronics, presents on the real-time GPU scheduling of XenGT in Automotive Embedded systems. It introduces the real-time GPU schedule of XenGT running on automotive embedded systems and explains why this should be used for an automotive system.


Xen Project is consistently being used within embedded systems for automotive. Earlier this year at CES, GlobalLogic showcased its technology behind Nautilus, which is the company’s virtualisation solution that enables multiple domains to share the GPU hardware with no more than a 5 percent overall in performance changes. More on this technology and how it uses Xen Project here.

The summit was a huge success with many interesting conversations. The Xen Project thanks everyone who attended and presented as well as the sponsors of the event Citrix, Huawei and Intel.

Xen Project 4.5.5 Maintenance Release is Available

I am pleased to announce the release of Xen 4.5.5. Xen Project Maintenance releases are released in line with our Maintenance Release Policy. We recommend that all users of the 4.5 stable series update to this point release.

Xen 4.5.5 is available immediately from its git repository:

    xenbits.xenproject.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.5
    (tag RELEASE-4.5.5)

or from the Xen Project download page at www.xenproject.org/downloads/xen-archives/xen-45-series/xen-455.html.

This release contains many bug fixes and improvements. For a complete list of changes in this release, please check the lists of changes on the download page.

We recommend all users of the 4.5 stable series to update to this latest point release.

PV Calls: a new paravirtualized protocol for POSIX syscalls

Let’s take a step back and look at the current state of virtualization in the software industry. X86 hypervisors were built to run a few different operating systems on the same machine. Nowadays they are mostly used to execute several instances of the same OS (Linux), each running a single server application in isolation. Containers are a better fit for this use case, but they expose a very large attack surface. It is possible to reduce the attack surface, however it is a very difficult task, one that requires minute knowledge of the app running inside. At any scale it becomes a formidable challenge. The 15-year-old hypervisor technologies, principally designed for RHEL 5 and Windows XP, are more a workaround than a solution for this use case. We need to bring them to the present and take them into the future by modernizing their design.

The typical workload we need to support is a Linux server application which is packaged to be self contained, complying to the OCI Image Format or Docker Image Specification. The app comes with all required userspace dependencies, including its own libc. It makes syscalls to the Linux kernel to access resources and functionalities. This is the only interface we must support.

Many of these syscalls closely correspond to function calls which are part of the POSIX family of standards. They have well known parameters and return values. POSIX stands for “Portable Operating System Interface”: it defines an API available on all major Unixes today, including Linux. POSIX is large to begin with and Linux adds its own set of non-standard calls on top of it. As a result a Linux system has a very high number of exposed calls and, inescapably, also a high number of vulnerabilities. It is wise to restrict syscalls by default. Linux containers struggle with it, but hypervisors are very accomplished in this respect. After all hypervisors don’t need to have full POSIX compatibility. By paravirtualizing hardware interfaces, Xen provides powerful functionalities with a small attack surface. But PV devices are the wrong abstraction layer for Docker apps. They cause duplication of functionalities between the guest and the host. For example, the network stack is traversed twice, first in DomU then in Dom0. This is unnecessary. It is better to raise hypervisor abstractions by paravirtualizing a small set of syscalls directly.

PV Calls

It is far easier and more efficient to write paravirtualized drivers for syscalls than to emulate hardware because syscalls are at a higher level and made for software. I wrote a protocol specification called PV Calls to forward POSIX calls from DomU to Dom0. I also wrote a couple of prototype Linux drivers for it that work at the syscall level. The initial set of calls covers socket, connect, accept, listen, recvmsg, sendmsg and poll. The frontend driver forwards syscalls requests over a ring. The backend implements the syscalls, then returns success or failure to the caller. The protocol creates a new ring for each active socket. The ring size is configurable on a per socket basis. Receiving data is copied to the ring by the backend, while sending data is copied to the ring by the frontend. An event channel per ring is used to notify the other end of any activity. This tiny set of PV Calls is enough to provide networking capabilities to guests.

We are still running virtual machines, but mainly to restrict the vast majority of applications syscalls to a safe and isolated environment. The guest operating system kernel, which is provided by the infrastructure (it doesn’t come with the app), implements syscalls for the benefit of the server application. Xen gives us the means to exploit hardware virtualization extensions to create strong security boundaries around the application. Xen PV VMs enable this approach to work even when virtualization extensions are not available, such as on top of Amazon EC2 or Google Compute Engine instances.

This solution is as secure as Xen VMs but efficiently tailored for containers workloads. Early measurements show excellent performance. It also provides a couple of less obvious advantages. In Docker’s default networking model, containers’ communications appear to be made from the host IP address and containers’ listening ports are explicitly bound to the host. PV Calls are a perfect match for it: outgoing communications are made from the host IP address directly and listening ports are automatically bound to it. No additional configurations are required.

Another benefit is ease of monitoring. One of the key aspects of hardening Linux containers is keeping applications under constant observation with logging and monitoring. We should not ignore it even though Xen provides a safer environment by default. PV Calls forward networking calls made by the application to Dom0. In Dom0 we can trivially log them and detect misbehavior. More powerful (and expensive) monitoring techniques like memory introspection offer further opportunities for malware detection.

PV Calls are unobtrusive. No changes to Xen are required as the existing interfaces are enough. Changes to Linux are very limited as the drivers are self-contained. Moreover, PV Calls perform extremely well! Let’s take a look at a couple of iperf graphs (higher is better):

iperf client

iperf server

The first graph shows network bandwidth measured by running an iperf server in Dom0 and an iperf client inside the VM (or container in the case of Docker). PV Calls reach 75 gbit/sec with 4 threads, far better than netfront/netback.

The second graph shows network bandwidth measured by running an iperf server in the guest (or container in the case of Docker) and an iperf client in Dom0. In this scenario PV Calls reach 55 gbit/sec and outperform not just netfront/netback but even Docker.

The benchmarks have been run on an Intel Xeon D-1540 machine, with 8 cores (16 threads) and 32 GB of ram. Xen is 4.7.0-rc3 and Linux is 4.6-rc2. Dom0 and DomU have 4 vcpus each, pinned. DomU has 4 GB of ram.

For more information on PV Calls, read the full protocol specification on xen-devel. You are welcome to join us and participate in the review discussions. Contributions to the project are very appreciated!

Virtual Machine Introspection: A Security Innovation With New Commercial Applications

The article from Lars Kurth, the Xen Project chairperson, was first published on Linux.com.

A few weeks ago, Citrix and Bitdefender launched XenServer 7 and Bitdefender Hypervisor Introspection, which together compose the first commercial application of the Xen Project Hypervisor’s Virtual Machine Introspection (VMI) infrastructure. In this article, we will cover why this technology is revolutionary and how members of the Xen Project Community and open source projects that were early adopters of VMI (most notably LibVMI and DRAKVUF) collaborated to enable this technology.

Evolving Security Challenges in Virtual Environments

Today, malware executes in the same context and with the same privileges as anti-malware software. This is an increasing problem, too. The Walking Dead analogy I introduced in this Linux.com article is again helpful. Let’s see how traditional anti-malware software fits into the picture and whether our analogy applies to anti-malware software.

In the Walking Dead universe, Walkers have taken over the earth, feasting on the remaining humans. Walkers are active all the time, and attracted by sound, eventually forming a herd that may overrun your defences. They are strong, but are essentially dumb. As we explored in that Linux.com article, people make mistakes, so we can’t always keep Walkers out of our habitat.

For this analogy, let’s equate Walkers with malware. Let’s assume our virtualized host is a village, consisting of individual houses (VMs) while the Hypervisor and network provides the infrastructure (streets, fences, electricity, …) that bind the village together.

Enter the world of anti-malware software: assume the remaining humans have survived for a while and re-developed technology to identify Walkers fast, destroy them quickly and fix any damage caused. This is the equivalent of patrols, CCTV, alarmed doors/windows and other security equipment, troops to fight Walkers once discovered and a clean-up crew to fix any damage. Unfortunately, the reality of traditional malware security technology can only be deployed within individual houses (aka VMs) and not on the streets of our village.

To make matters worse, until recently malware was relatively dumb. However, this has changed dramatically in the last few years. Our Walkers have evolved into Wayward Pine’s Abbies, which are faster, stronger and more intelligent than Walkers. In other words, malware is now capable of evading or disabling our security mechanisms.

What we need is the equivalent of satellite surveillance to observe the entire village, and laser beams to remotely destroy attackers when they try and enter our houses. We can of course also use this newfound capability to quickly deploy ground troops and clean-up personnel as needed. In essence that is the promise that Virtual Machine Introspection gives us. It allows us to address security issues from outside the guest OS without relying on functionality that can be rendered unreliable from the ground. More on that topic later.

From VMI in Xen to the First Commercial Application: A Tale of Collaboration

The development of Virtual Machine Introspection and its applications show how the Xen Project community is bringing revolutionary technologies to market.

The development of Virtual Machine Introspection and its applications show how the Xen Project community is bringing revolutionary technologies to market.

The idea of Virtual Machine Introspection for the Xen Project Hypervisor hatched at Georgia Tech in 2007, building on research by Tal Garfinkel and Mendel Rosenblum in 2003. The technology was first incorporated into the Xen Project Hypervisor via the XenAccess and mem-events APIs in 2009. To some degree, this was a response to VMware’s VMsafe technology, which was introduced in 2008 and deprecated in 2012, as the technology had significant limitations at scale. VMSafe was replaced by vShield, which is an agent-based, hypervisor-facilitated, file-system anti-virus solution that is effectively a subset of VMsafe.

Within the Xen Project software however, Virtual Machine Introspection technology lived on due to strong research interests and specialist security applications where trading off performance against security was acceptable. This eventually led to the creation of LibVMI (2010), which made these APIs more accessible. This provided an abstraction that eventually allowed exposure of a subset of Xen’s VMI functionality to other open source virtualization technologies such as KVM and QEMU.

In May 2013, Intel launched its Haswell generation of CPUs, which is capable of maintaining up to 512 EPT pointers from the VMCS via the #VE and VMFUNC extensions. This proved to be a potential game-changer for VMI, enabling hypervisor controlled and hardware enforced strong isolation between VMs with lower than previous overheads, which led to a collaboration of security researchers and developers from Bitdefender, Cisco, Intel, Novetta, TU Munich and Zentific. From 2014 to 2015, the XenAccess and mem-events APIs have been re-architected into the Xen Project Hypervisor’s new VMI subsystem, alt2pm and other hardware capabilities have been added, as well as support for ARM CPUs and a baseline that was production ready has been released in Xen 4.6.

Citrix and Bitdefender collaborated to bring VMI technology to market: XenServer 7.0 introduced its Direct Inspect APIs built on the Xen Projects VMI interface. It securely exposes the introspection capabilities to security appliances, as implemented by Bitdefender HVI.

What Can Actually Be Done Today?

Coming back to our analogy: what we need is the equivalent of satellite surveillance to observe the entire village. Does VMI deliver? In theory, yes: VMI makes it possible to observe the state of any virtual machine (house and its surroundings in the village), including memory and CPU state and to receive events when the state of the virtual machine changes (aka if there is any movement). In practice, the performance overhead of doing this is far too high, despite using hardware capabilities.

In our imagined world that is overrun by Walkers and Abbies, this is equivalent to not having the manpower to monitor everything, which means we have to use our resources to focus on high value areas. In other words, we need to focus on the suspicious activity on system perimeters (the immediate area surrounding each of our houses).

This focus is executed by monitoring sensitive memory areas for suspicious activity. When malicious activity is detected, a solution can take corrective actions on the process state (block, kill) or VM state (pause, shutdown) while collecting and reporting forensic details directly from a running VM.

Think of a laser beam on our satellite that is activated whenever an Abbie or Walker approaches our house. In technical terms, the satellite and laser infrastructure maps to XenServer’s Direct Inspect API, while the software which controls and monitors our data maps onto Bitdefenders Hypervisor Introspection.

It is important to stress that monitoring and remedial action takes place from the outside, using the hypervisor to provide hardware-enforced isolation. This means that our attackers cannot disable surveillance nor laser beams.

Of course, no security solution is perfect. This monitoring software may not always detect all suspicious activity, if that activity does not impact VM memory. This does not diminish the role of file-system-based security; we must still be vigilant, and there is no perfect defense. In our village analogy, we could also be attacked through underground infrastructure such as tunnels and canalisation. In essence this means we have to use VMI together with traditional anti-malware software.

How does VMI compare to traditional hypervisor-facilitated anti-virus solutions such as vShield? In our analogy, these solutions require central management of all surveillance equipment that is installed in our houses (CCTV, alarmed doors/windows, …) while the monitoring of events is centralized very much like a security control centre in our village hall. Albeit such an approach significantly simplifies monitoring and managing of what goes on within virtual machines, it does not deliver the extra protection that introspection provides.

You can find more information (including some demos) about VMI, XenServer Direct Inspect API and BitDefender Hypervisor Introspection here:

Xen Project Virtual Machine Introspection

Conclusion

The development of VMI and its first open source and commercial applications show how the Xen Project community is innovating in novel ways, and is capable of bringing revolutionary technologies to market. The freedom to see the code, to learn from it, to ask questions and offer improvements has enabled security researchers and vendors such as Citrix and Bitdefender to bring new solutions to market.

It is also worth pointing out that hardware-enabled security technology is moving very fast: only a subset of Intel’s #VE and VMFUNC extensions are currently being deployed in VMI. Making use of more hardware extensions carries the promise of combining the protection of out-of-guest tools with the performance of in-guest tools.

What is even more encouraging is that other vendors such as A1Logic, Star Lab and Zentific are working on new Xen Project-based security solutions. In addition, the security focused, Xen-based OpenXT project has started to work more closely with the Xen Project community, which promises further security innovation.

A few of these topics will be discussed in more detail during Xen Project Developer Summit happening in Toronto, CA from August 25 – 26, 2016. You learn more about the event here.

Don’t Miss Xen Project & KVM Joint Reception at LinuxCon

Last year, we really enjoyed co-hosting a hackathon and social event with the KVM community. It spurred really interesting conversations, a bit of friendly competition and some community bonding.

Back by popular demand is another joint KVM community event. Xen Project and KVM are hosting the joint social event at the Hockey Hall of Fame (this is Canada after all) on August 25th during the Xen Project Developer Summit and KVM Forum. The event will be held from 7:00pm to 11:00pm.

xenparty2This will be a great evening for anyone who is attending the Xen Project Developer Summit and KVM Forum, August 25 – 26.

xenparty

The Hockey Hall of Fame is a ten minute walk from the Westin where LinuxCon and ContainerCon are taking place. There will be plenty of shop talk of course. The Xen Project is increasingly more popular in IoT, automobile and embedded use cases, and a staple open source software in many of the largest companies today.

But there will also be plenty of time to check out the interactive hockey games, amazing hockey memorabilia, food, and drinks. Your badge is required to enter the party.

If you are interested in joining us for the Xen Project and KVM party, you must be attending one of these events. If you haven’t already, registration for the Xen Project Developer Summit is here. A few more highlights include:

  • Porting Xen on ARM to a new SOC with Julien Grall of ARM

  • High-Performance Virtualization for HPC Cloud on Xen with Tianyu Lan and Jun Nakajima of Intel

  • Attack Surface Reduction with Douglas Goldstein of Star Labs

  • Patch Review for Non-Maintainers with George Dunlap of Citrix

  • Xen Scalability Analysis with Weidong Han, Zichao Huang, and Wei Yang of Huawei

Q&A: Xen Project Release Strengthens Security and Pushes New Use Cases

The following Q&A with Lars Kurth, the Xen Project chairperson, was first published on Linux.com.

Xen Project technology supports more than 10 million users and is a staple in some of the largest clouds in production today, including Amazon Web Service, Tencent, and Alibaba’s Aliyun. Recently, the project announced the arrival of Xen Project Hypervisor 4.7. This new release focuses on improving code quality, security hardening and features, and support for the latest hardware. It is also the first release of the project’s fixed-term June – December release cycles. The fixed-term release cycles provide more predictability making it easier for consumers of Xen to plan ahead.

We recently sat down with the Xen Project chairperson, Lars Kurth, to talk about some of the key features of the release and the future of Xen Project technology. Lars will be discussing this topic and more during Xen Project’s Developer Summit in Toronto, CA from August 25-26 — the conference is directly after LinuxCon North America.

Q: What was the focus on this release?

Lars Kurth: There were five areas that we focused on for this release (full details are in our blog). In summary, we focused on security features, migration support, performance and workloads, support for new hardware features, and drivers and devices (Linux, FreeBSD and other).

Security is consistently something that we focus on in all of our releases. There are a lot of people that rely on Xen Project technology and security is our top concern in any release as well as how we organize our process around security disclosures.

Q: What was the biggest feature coming out of this release?

Lars: The biggest feature for us is live patching, which is a technology that enables re-boot free deployment for security patches to minimize disruption and downtime during security upgrades for cloud admins. It essentially eliminates all cloud reboots, making cloud providers and their users much more safe. It also eliminates a lot of headaches for system and DevOps admins of the world.

Q: Xen is often associated with the cloud, but are there additional use cases that you see growing around this technology, if so why?

Lars: We are seeing a lot of growth in terms of contributions, as well as many different use cases emerging, including automotive, aviation, embedded scenarios, security, and also IoT. In addition, we continue to grow within the public cloud sector and traditional server virtualization.

On the security front, for example, a number of vendors such as A1Logic, Bitdefender, Star Lab and Zentific have released or are working on new Xen Project-based security solutions. In addition, the security focused and Xen-based OpenXT project has started to work more closely with the Xen Project community.

Long-time contributors to the Xen Project, such as DornerWorks – a premier provider of electronic engineering services for the aerospace, medical, automotive, and industrial markets – have expanded their scope and are now providing support for the Xen Xilinx Zynq Distribution targeting embedded use-cases. We have also seen an increasing number of POCs and demos of automotive solutions, which include Xen as a virtualization solution.

Growth in these sectors is largely due to the Xen Project’s flexibility, extensibility, customisability and a clear lead when it comes to security-related technologies. Over the last year, we have also seen contributions increase from developers with strong security and embedded backgrounds. In fact, this totaled nearly 17 percent of the overall contributions in this release cycle, up from 9 percent in the previous release.

Q: How did you address these uses cases in this latest release?

Lars: We introduced the ability to remove core Xen Project Hypervisor features at compile via KCONFIG. This creates a more lightweight hypervisor and eliminates extra attack surfaces that are beneficial in security-first environments and microservice architectures. Users will still be able to get the core hypervisor functions, but they won’t receive all the drivers, schedulers, components or features that might not fit their use case.

Essentially it gives people an “a la carte” feature set. They can decide what they need for compliance, safety or performance reasons.

Q: Were there any new contributors for this release that surprised you?

Lars: We had three new companies contributing to the project: Star Lab, Bosch and Netflix. I met engineers from Star Lab for the first time at the 2015 Developer Summit less than a year ago, and helped introduce them to the Project’s culture. In that short period of time, Doug Goldstein from Star Lab has moved into the top five contributors and top 10 code reviewers for the Project.

I was surprised about Netflix’s contributions; I didn’t even know the company used Xen. Netflix improved and secured the VPMU feature, which is incredibly useful for system tuning and performance monitoring. Bosch Car Multimedia GmbH added some new ARM functionality. In addition, we have seen quite a bit of Xen related development in upstream and downstream projects such as Linux, FreeBSD, NetBSD, OpenBSD, QEMU and Libvirt.

Q: What’s next for Xen Project? Where do you think the technology is heading in the future and why?

Lars: In the last three releases, we introduced several major new features such as PVH, COLO, new schedulers, VMI, Live Patching, Graphics Virtualization, etc. and significant re-work of existing features such as Migration and the Xen Security Modules (XSM). Looking at trends within the community, I expect that stepwise evolution of large new features to continue.

Some new capabilities, such as restartable Dom0’s, and additional techniques to provide more isolation and security, are also likely to appear. In addition, it looks likely that we will see some GPU virtualization capabilities for GPUs that target the ARM ecosystem, although it is not yet clear whether these will be available as open source. I also expect that both Intel and ARM hardware features will be closely tracked.

Some areas, such as new schedulers, XSM, PVH and Live Patching, will see significant efforts to harden and improve existing functionality. The goal is to ensure their swift adoption in commercial products and Linux and BSD distributions. Some features, which are not enabled by default are likely to become part of the Xen Project Hypervisor’s default configuration.