Author Archives: Lars Kurth

About Lars Kurth

Lars Kurth is a highly effective, passionate community manager with strong experience of working with open source communities (Symbian, Symbian DevCo, Eclipse, GNU) and currently is community manager for xen.org. Lars has 9 years of experience building and leading engineering teams and a track record of executing several change programs impacting 1000 users. Lars has 16 years of industry experience in the tools and mobile sector working at ARM, Symbian Ltd, Symbian Foundation and Nokia. Lars has strong analytical, communication, influencing and presentation skills, good knowledge of marketing and product management and extensive background in C/C , Java and software development practices which he learned working as community manager, product manager, chief architect, engineering manager and software developer. If you want to know more, check out uk.linkedin.com/in/larskurth. Personally, Lars has a wide range of interests such as literature, theatre, cinema, cooking and gardening. He is particularly fascinated by orchids and carnivorous plants and has built a rather large collection of plants from all over the world. His love for plants extends into a passion for travel, in particular to see plants grow in their native habitats.

Now Accepting Submissions for Xen Project Developer and Design Summit 2017

screen-shot-2017-03-14-at-17-02-17 
We’re excited to announce that registration and the call for proposals is open for Xen Project Developer and Design Summit 2017, which will be held in Budapest, Hungary from July 11-13, 2017. The Xen Project Developer and Design Summit combines the formats of Xen Project Developer Summits with Xen Project Hackathons, and brings together the Xen Project’s community of developers and power users.

Submit a Talk

Do you have an interesting use case around Xen Project technology or best practices around the community? There’s a wide variety of topics we are looking for, including security, embedded environments, network function virtualization (NFV), and more. You can find all the suggested topics for presentations and panels here (make sure you select the Topics tab).

Several formats are being accepted for speaking proposals, including:

  • Presentations and Panels
  • Interactive design and problem solving sessions. These sessions can be submitted as part of the CFP, but we will reserve a number of design sessions to be allocated during the event. Proposers of design sessions are expected to host and moderate design sessions following the format we have used at Xen Project Hackathons. If you have not participated in these in the past, check out past event reports from 2016, 2015 and 2013.

Never talked at a conference before? Don’t worry! We encourage new speakers to submit for our events and have plenty of resources to help you prepare for your presentation.

Here are some dates to remember for submissions and in general:

  • CFP Close: April 14, 2017
  • CFP Notifications: May 5, 2017
  • Schedule Announced: May 16, 2017
  • Event: July 11-13, 2017

Registration

Come join us for this event, and if you register by May 19, you’ll get an early bird discount :) Travel stipends are available for students or individuals that are not associated with a company. If you have any questions, please send a note to community.manager@xenproject.org.

Xen Project Participates in Google Summer of Code and Outreachy

This is a quick announcement that the Xen Project is again participating in Google Summer of Code (GSoC), a program that awards three-month paid stipends to University students to work on open source projects, with the goal to get open source experience. The Xen Project will also again participate in Outreachy, which is an internship program that organises three-month paid internships with free and open-source software projects for people who are typically underrepresented in the technology industry. Outreachy has been helping women (cis and trans), trans men, genderqueer people and people from other discriminated backgrounds get involved in free and open source software for several years. The Xen Project is proud that it has participated continually in Outreachy (and its predecessor OPW) for 4 years.

I want to participate, how do I get started?

If you are not at all familiar with programs such as GSoC and Outreachy, have a quick look at our introduction. In a nutshell, both programs go through several stages:

  • Check eligibility requirements.
  • Now until application period: Preparation by working on small tasks (also called micro-tasks) within our community to identify a suitable project and to familiarise yourself with the technology.
  • Application Period (aka paperwork): For GSoC, the application system is open from March 20 to Apr 3, 2017; however you should work on micro-tasks before and prepare your application together with a mentor as early as possible. For Outreachy, the application system is already open and will close March 30, 2017 (but you can edit and modify proposals submitted in agreement with your mentor until April 28th).
  • Selection Period: After applying to participate, our mentors will chose the most promising candidates. Successful candidates will be announced on the following dates: April 28 (Outreachy), May 4 (GSoC).
  • Internship Duration: May 30 to August 29 (GSoC) and August 30 (Outreachy).

For a list of projects for participants and more information on how to apply, check our Xen Project 2017 Summer Internship Portal. We have many different projects in many different areas: from Hypervisor work, projects in Mirage OS, to tools and test related tasks. Note that we will be adding extra projects to this page in the coming weeks and that applicants can suggest projects on their own.

You may also want to check out the pages of GSoC mentoring organisations which we collaborate with. Sometimes, you will find Xen related projects there: FreeBSD (currently 2 projects), QEMU, Libvirt.

Learn about the Experience of past Participants

At a past Xen Project Developer Summit, we ran a panel discussion that included Outreachy interns, GSoC students as well as mentors.


You may also want to read Women interns rocking open source at Xen Project.

What You Need to Know about Recent Xen Project Security Advisories

Today the Xen Project announced eight security advisories: XSA-191 to XSA-198. The bulk of these security advisories were discovered and fixed during the hardening phase of the Xen Project Hypervisor 4.8 release (expected to come out in early December). The Xen Project has implemented a security-first approach when publishing new releases.

In order to increase the security of future releases, members of the Xen Project Security Team and key contributors to the Xen Project, actively search and fix security bugs in code areas where vulnerability were found in past releases. The contributors use techniques such as code inspections, static code analysis, and additional testing using fuzzers such as American Fuzzy Lop. These fixes are then backported to older Xen Project releases with security support and published in bulk to make it easier for downstreams consumers to apply security fixes.

Before we declare a new Xen Project feature as supported, we perform a security assessment (see declare the Credit2 scheduler as supported). In addition, the contributors focused on security have started crafting tests for each vulnerability and integrated them into our automated regression testing system run regularly on all maintained versions of Xen. This ensures that the patch will be applied to every version which is vulnerable, and also ensures that no bug is accidentally reintroduced as development continues to go forward.

The Xen Project’s mature and robust security response process is optimized for cloud environments and downstream Xen Project consumers to maximize fairness, effectiveness and transparency. This includes not publicly discussing any details with security implications during our embargo period. This encourages anyone to report bugs they find to the Xen Project Security team, and allows the Xen Project Security team to assess, respond, and prepare patches, before before public disclosure and broad compromise occurs.

During the embargo period, the Xen Project does not publicly discuss any details with security implications except:

  • when co-opting technical assistance from other parties;
  • when issuing a Xen Project Security Advisory (XSA). This is pre-disclosed to only members on the Xen Project Pre-Disclosure List (see www.xenproject.org/security-policy.html); and
  • when necessary to coordinate with other projects affected

The Xen Project security team will assign and publicly release numbers for vulnerabilities. This is the only information that is shared publicly during the embargo period. See this url for “XSA Advisories, Publicly Released or Pre-Released”: xenbits.xen.org/xsa.

Xen’s latest XSA-191, XSA-192, XSA-193, XSA-194, XSA-195, XSA-196, XSA-197 and XSA-198 Advisory can all be found here:
xenbits.xen.org/xsa

Any Xen-based public cloud is eligible to be on our “pre-disclosure” list. Cloud providers on the list were notified of the vulnerability and provided a patch two weeks before the public announcement in order to make sure they all had time to apply the patch to their servers.

Distributions and other major software vendors of Xen Project software were also given the patch in advance to make sure they had updated packages ready to download as soon as the vulnerability was announced. Private clouds and individuals are urged to apply the patch or update their packages as soon as possible.

All of the above XSAs that affect the hypervisor can be deployed using the Xen Project LivePatching functionality, which enables re-boot free deployment of security patches to minimize disruption and downtime during security upgrades for system administrators and DevOps practitioners. The Xen Project encourages its users to download these patches.

More information about the Xen Project’s Security Vulnerability Process, including the embargo and disclosure schedule, policies around embargoed information, information sharing among pre-disclosed list members, a list of pre-disclosure list members, and the application process to join the list, can be found at: www.xenproject.org/security-policy.html

Announcing Xen Project 4.8 RC and Test Day Schedule

On Monday, we created Xen 4.8 RC1 and will release a new release candidate every week, until we declare a release candidate as the final candidate and cut the Xen 4.8 release. We will also hold a Test Day every Friday for the release candidate that was released the week prior to the Test Day. Note that RC’s are announced on the following mailing lists: xen-announce, xen-devel and xen-users. This means we will have Test Days on October 7th, 14th, 21st and 28th.

Your testing is still valuable on other days, so please feel free to send Test Reports as outlined below at any time.

Getting, Building and Installing a Release Candidate

Release candidates are available from our git repository at

git://xenbits.xenproject.org/xen.git (tag 4.8.0-<rc>)

where <rc> is rc1, rc2, rc3, etc. and as tarball from

https://downloads.xenproject.org/release/xen/4.8.0-<rc>/xen-4.8.0-<rc>.tar.gz
https://downloads.xenproject.org/release/xen/4.8.0-<rc>/xen-4.8.0-<rc>.tar.gz.sig

Detailed build and Install instructions can be found on the Test Day Wiki.

Testing new Features, Test and Bug Reports

You can find Test Instructions for new features on our Test Day Wiki and instructions for general tests on Testing Xen. The following pages provide information on how to report successful tests and how to report bugs and issues.

Happy Testing!

A Recap of the Xen Project Developer Summit 2016

The Xen Project descended on Toronto, Canada in late August for its annual Xen Project Developer Summit. The Summit is an opportunity for developers and software engineers to collaborate and discuss the latest advancements of the Xen Project software. It also gives developers a chance to better understand new trends and deployments in the community and from power enterprise users.

From community growth to new emerging use cases, the Summit covered a lot of ground. Developments within core technologies such as security, graphics support and hardware support were discussed. We also covered emerging technologies such as automotive, embedded and IoT. All sessions were recorded and are available here and also on slideshare (follow this link for summit presentations).

Below is a summary of a few videos that feature technology that has been recently introduced into the Xen Project hypervisor as well as emerging technologies that are being built with Xen Project technology.

New Feature Technologies from Xen Project Community and Power Users

In Xen Project 4.7, we introduced Live Patching as a technology preview. Live Patching gives system administrators and DevOps practitioners the ability to update the Xen Project hypervisor without the need for a reboot. Konrad Wilk, software development manager of Oracle and Ross Lagerwall, software enggineer at Citrix, provide insight into how it works, what the difficulties were to implement, and how it compares to other technologies for patching (kGraft, kPatch, kSplice, Linux hot-patching).



Dimitri Stilliadis, CEO of Aporeto, provides a great overview of the benefits of using Xen Project software to provide an execution environment for Docker apps. This approach allows VM-like isolations for security measures without having to sacrifice performance. The presentation introduces a new paravirtualized protocol to virtualise IP sockets and provides the design and implementation details.


Data breaches are happening all the time, and there are many ways that organisations are trying to stop this through detection, pattern matching and behavioural analysis. However, Neil Sikka, founder and CEO of A1LOGIC, provides a new way of looking at this problem and solving this problem by using the Xen Project hypervisor to enforce data loss prevention. It doesn’t use any type of detection, heuristics, pattern matching or behavioural analysis, but rather a strictly algorithmic approach rooted in hardware.

Embedded Projects and Xen Project Software

Members from the Xen Project sister community OpenXT, an open-source development toolkit for hardware-assisted security research and appliance integration, were present to provide some insights into how Xen Project is working within the embedded space and best practices for embedding Xen Project on mobile and tablet devices.

If this is an area that you are interested in, check out Christopher Clark (consultant and interoperability architect at BAE Systems) overview of the OpenXT Project, which has begun to attract new users and contributors. We also recommend Chris Patterson’s and Kyle Temkin’s step-by-step guide on the challenges and lessons to get Xen Project software started on phones and tablets. Chris is a advising computer engineer for AIS and Kyle is researcher for AIS.


Emerging Technologies

Xen Project is consistently becoming more common within automotive and aviation. Xen Project 4.7 introduced the ability to remove core Xen Hypervisor features at compile time via KCONFIG. This allows a more lightweight hypervisor, which is perfect for IoT scenarios and better for security-first environments, like automotive.

Sangyun Lee, senior embedded software engineer of LG Electronics, presents on the real-time GPU scheduling of XenGT in Automotive Embedded systems. It introduces the real-time GPU schedule of XenGT running on automotive embedded systems and explains why this should be used for an automotive system.


Xen Project is consistently being used within embedded systems for automotive. Earlier this year at CES, GlobalLogic showcased its technology behind Nautilus, which is the company’s virtualisation solution that enables multiple domains to share the GPU hardware with no more than a 5 percent overall in performance changes. More on this technology and how it uses Xen Project here.

The summit was a huge success with many interesting conversations. The Xen Project thanks everyone who attended and presented as well as the sponsors of the event Citrix, Huawei and Intel.

Virtual Machine Introspection: A Security Innovation With New Commercial Applications

The article from Lars Kurth, the Xen Project chairperson, was first published on Linux.com.

A few weeks ago, Citrix and Bitdefender launched XenServer 7 and Bitdefender Hypervisor Introspection, which together compose the first commercial application of the Xen Project Hypervisor’s Virtual Machine Introspection (VMI) infrastructure. In this article, we will cover why this technology is revolutionary and how members of the Xen Project Community and open source projects that were early adopters of VMI (most notably LibVMI and DRAKVUF) collaborated to enable this technology.

Evolving Security Challenges in Virtual Environments

Today, malware executes in the same context and with the same privileges as anti-malware software. This is an increasing problem, too. The Walking Dead analogy I introduced in this Linux.com article is again helpful. Let’s see how traditional anti-malware software fits into the picture and whether our analogy applies to anti-malware software.

In the Walking Dead universe, Walkers have taken over the earth, feasting on the remaining humans. Walkers are active all the time, and attracted by sound, eventually forming a herd that may overrun your defences. They are strong, but are essentially dumb. As we explored in that Linux.com article, people make mistakes, so we can’t always keep Walkers out of our habitat.

For this analogy, let’s equate Walkers with malware. Let’s assume our virtualized host is a village, consisting of individual houses (VMs) while the Hypervisor and network provides the infrastructure (streets, fences, electricity, …) that bind the village together.

Enter the world of anti-malware software: assume the remaining humans have survived for a while and re-developed technology to identify Walkers fast, destroy them quickly and fix any damage caused. This is the equivalent of patrols, CCTV, alarmed doors/windows and other security equipment, troops to fight Walkers once discovered and a clean-up crew to fix any damage. Unfortunately, the reality of traditional malware security technology can only be deployed within individual houses (aka VMs) and not on the streets of our village.

To make matters worse, until recently malware was relatively dumb. However, this has changed dramatically in the last few years. Our Walkers have evolved into Wayward Pine’s Abbies, which are faster, stronger and more intelligent than Walkers. In other words, malware is now capable of evading or disabling our security mechanisms.

What we need is the equivalent of satellite surveillance to observe the entire village, and laser beams to remotely destroy attackers when they try and enter our houses. We can of course also use this newfound capability to quickly deploy ground troops and clean-up personnel as needed. In essence that is the promise that Virtual Machine Introspection gives us. It allows us to address security issues from outside the guest OS without relying on functionality that can be rendered unreliable from the ground. More on that topic later.

From VMI in Xen to the First Commercial Application: A Tale of Collaboration

The development of Virtual Machine Introspection and its applications show how the Xen Project community is bringing revolutionary technologies to market.

The development of Virtual Machine Introspection and its applications show how the Xen Project community is bringing revolutionary technologies to market.

The idea of Virtual Machine Introspection for the Xen Project Hypervisor hatched at Georgia Tech in 2007, building on research by Tal Garfinkel and Mendel Rosenblum in 2003. The technology was first incorporated into the Xen Project Hypervisor via the XenAccess and mem-events APIs in 2009. To some degree, this was a response to VMware’s VMsafe technology, which was introduced in 2008 and deprecated in 2012, as the technology had significant limitations at scale. VMSafe was replaced by vShield, which is an agent-based, hypervisor-facilitated, file-system anti-virus solution that is effectively a subset of VMsafe.

Within the Xen Project software however, Virtual Machine Introspection technology lived on due to strong research interests and specialist security applications where trading off performance against security was acceptable. This eventually led to the creation of LibVMI (2010), which made these APIs more accessible. This provided an abstraction that eventually allowed exposure of a subset of Xen’s VMI functionality to other open source virtualization technologies such as KVM and QEMU.

In May 2013, Intel launched its Haswell generation of CPUs, which is capable of maintaining up to 512 EPT pointers from the VMCS via the #VE and VMFUNC extensions. This proved to be a potential game-changer for VMI, enabling hypervisor controlled and hardware enforced strong isolation between VMs with lower than previous overheads, which led to a collaboration of security researchers and developers from Bitdefender, Cisco, Intel, Novetta, TU Munich and Zentific. From 2014 to 2015, the XenAccess and mem-events APIs have been re-architected into the Xen Project Hypervisor’s new VMI subsystem, alt2pm and other hardware capabilities have been added, as well as support for ARM CPUs and a baseline that was production ready has been released in Xen 4.6.

Citrix and Bitdefender collaborated to bring VMI technology to market: XenServer 7.0 introduced its Direct Inspect APIs built on the Xen Projects VMI interface. It securely exposes the introspection capabilities to security appliances, as implemented by Bitdefender HVI.

What Can Actually Be Done Today?

Coming back to our analogy: what we need is the equivalent of satellite surveillance to observe the entire village. Does VMI deliver? In theory, yes: VMI makes it possible to observe the state of any virtual machine (house and its surroundings in the village), including memory and CPU state and to receive events when the state of the virtual machine changes (aka if there is any movement). In practice, the performance overhead of doing this is far too high, despite using hardware capabilities.

In our imagined world that is overrun by Walkers and Abbies, this is equivalent to not having the manpower to monitor everything, which means we have to use our resources to focus on high value areas. In other words, we need to focus on the suspicious activity on system perimeters (the immediate area surrounding each of our houses).

This focus is executed by monitoring sensitive memory areas for suspicious activity. When malicious activity is detected, a solution can take corrective actions on the process state (block, kill) or VM state (pause, shutdown) while collecting and reporting forensic details directly from a running VM.

Think of a laser beam on our satellite that is activated whenever an Abbie or Walker approaches our house. In technical terms, the satellite and laser infrastructure maps to XenServer’s Direct Inspect API, while the software which controls and monitors our data maps onto Bitdefenders Hypervisor Introspection.

It is important to stress that monitoring and remedial action takes place from the outside, using the hypervisor to provide hardware-enforced isolation. This means that our attackers cannot disable surveillance nor laser beams.

Of course, no security solution is perfect. This monitoring software may not always detect all suspicious activity, if that activity does not impact VM memory. This does not diminish the role of file-system-based security; we must still be vigilant, and there is no perfect defense. In our village analogy, we could also be attacked through underground infrastructure such as tunnels and canalisation. In essence this means we have to use VMI together with traditional anti-malware software.

How does VMI compare to traditional hypervisor-facilitated anti-virus solutions such as vShield? In our analogy, these solutions require central management of all surveillance equipment that is installed in our houses (CCTV, alarmed doors/windows, …) while the monitoring of events is centralized very much like a security control centre in our village hall. Albeit such an approach significantly simplifies monitoring and managing of what goes on within virtual machines, it does not deliver the extra protection that introspection provides.

You can find more information (including some demos) about VMI, XenServer Direct Inspect API and BitDefender Hypervisor Introspection here:

Xen Project Virtual Machine Introspection

Conclusion

The development of VMI and its first open source and commercial applications show how the Xen Project community is innovating in novel ways, and is capable of bringing revolutionary technologies to market. The freedom to see the code, to learn from it, to ask questions and offer improvements has enabled security researchers and vendors such as Citrix and Bitdefender to bring new solutions to market.

It is also worth pointing out that hardware-enabled security technology is moving very fast: only a subset of Intel’s #VE and VMFUNC extensions are currently being deployed in VMI. Making use of more hardware extensions carries the promise of combining the protection of out-of-guest tools with the performance of in-guest tools.

What is even more encouraging is that other vendors such as A1Logic, Star Lab and Zentific are working on new Xen Project-based security solutions. In addition, the security focused, Xen-based OpenXT project has started to work more closely with the Xen Project community, which promises further security innovation.

A few of these topics will be discussed in more detail during Xen Project Developer Summit happening in Toronto, CA from August 25 – 26, 2016. You learn more about the event here.