Author Archives: Lars Kurth

About Lars Kurth

Lars Kurth is a highly effective, passionate community manager with strong experience of working with open source communities (Symbian, Symbian DevCo, Eclipse, GNU) and currently is community manager for xen.org. Lars has 9 years of experience building and leading engineering teams and a track record of executing several change programs impacting 1000 users. Lars has 16 years of industry experience in the tools and mobile sector working at ARM, Symbian Ltd, Symbian Foundation and Nokia. Lars has strong analytical, communication, influencing and presentation skills, good knowledge of marketing and product management and extensive background in C/C , Java and software development practices which he learned working as community manager, product manager, chief architect, engineering manager and software developer. If you want to know more, check out uk.linkedin.com/in/larskurth. Personally, Lars has a wide range of interests such as literature, theatre, cinema, cooking and gardening. He is particularly fascinated by orchids and carnivorous plants and has built a rather large collection of plants from all over the world. His love for plants extends into a passion for travel, in particular to see plants grow in their native habitats.

A Recap of the Xen Project Developer and Design Summit: Community Health, Development Trends, Coding Changes and More

We were extremely thrilled to host our Xen Project Developer and Design Summit in Nanjing Jiangning, China this June. The event brought together our community and power users under one roof to collaborate and to learn more about the future of our project. It also gave us the opportunity to connect with a large group of our community who is based in China. We’ve seen a steady stream of Xen Project hypervisor adoption in this region.

If you were unable to attend the event, we have recordings of the presentations, and we also have the slideshares from the presentation available. Please check them out!

During our event, we always start with a weather report on the Xen Project. It covers areas that we are improving upon, where we need more support, and also the potential direction of the project. This blog covers information from the weather report as well as next steps and focus areas for the project.

Community Health
Code commits for the hypervisor have on average grown by 11% YoY since 2014. Commits in the first 5 months of 2018 have grown 11% compared to the same period last year. The top 6 contributors to the project since 2011 have been Citrix, Suse, AMD, Arm, Intel, Oracle. This is also true for the last 12 months in which 90% of contributions came from the top 6 players.

However, we have seen a larger than normal volume of contributions from Arm and AMD, which contributed twice as much as in previous years. In addition, EPAM is establishing itself on the top table with first contributions and a significant number of code reviews.  In addition, AWS started to make first significant code contributions in 2017.

Hardware security issues had an impact on the code review process of the project and thus on the project’s capability to take in some code. In other words, x86 related development that was not directly attributed to hardware security issues were slowed down, because developers normally reviewing contributions had less bandwidth to do so.

This has forced the community to make some changes that are starting to have a positive effect: x86 developers across companies are collaborating more and better, meaning that hardware security issues in 2018 had a smaller impact on the community than those in 2017.

Innovation and Development Trends
Unikraft, a Xen Project sub-project, is on a healthy growth projection. Unikraft aims to simplify the process of building unikernels through a unified and customizable code base. It was created after Xen Project Developer and Design Summit 2017.

The project recently upstreamed a significant amount of functionality, including:

  • Scheduling support, better/more complete support for KVM/Xen/Linux. Supporting Xen/KVM allows Unikraft to cater to a larger set of potential users/companies. Linux user-space provides an excellent development environment: Unikraft users can create their Unikraft unikernels as a Linux executable, use Linux’s wide range of debugging and performance optimization tools, and when done simply re-compile as a KVM or Xen unikernel (work on creating x86/Arm bare metal images is ongoing).
  • A release of newlib (a libc-like library) and lwip (a network stack: This support allows Unikraft to compile with most applications. It is a basic requirement to support a potentially wide range of applications.
  • The project is beginning to pick up traction with contributions coming from companies like NEC, Arm, and Oracle.

For more information check out the following two presentations: Unikraft and Unikraft on Arm.

We have been re-writing the x86 core. We are working on adding complex new CPU hardware features such as support for NVDIMMs and SGX. In addition, we are working on making technologies that have been used by security-conscious vendors in non-server environments ready to be used in server virtualization and cloud computing; support for measured boot is an example.

Another key innovation is a project called Panopticon, which aims to re-write some portions of the hypervisor to make Xen resilient to all types of side-channel attacks by removing unnecessary information about guests from the hypervisor.

You can find presentations related to these topics here (x86 evolution) and here (side-channel attacks and mitigations).

Continued Growth in Embedded and Automotive
We are seeing continued contributions within the embedded and automotive space to Xen Project Core with new features and functionality, including:

  • Co-processor (GPU) sharing framework enabling virtualization of co-processors such as FPGAs, DRMs, etc.
  • 2nd generation Power management and HPM on Arm  – this enables a huge reduction in power consumption, which is significant for some embedded market segments.
  • RTOS based Dom0 and code size reduction – this reduces the cost of safety certification significantly and is important for market segments where safety certification is important (such as automotive, avionics, medical, etc). We already managed to get Xen code size on Arm to below 45K SLOC and we expect that Dom0 will also be below 50K SLOC. This makes it possible to safety certify a Xen based stack to DAL C ASIL-B/C standards at a cost equivalent to less than 10 years.
  • Improved startup latency to boot multiple VMs in parallel from the device tree – this opens up the use of Xen to small IoT and embedded devices and allows booting of a complete Xen system in milliseconds compared to seconds. In addition, it halves the cost of safety certifications for systems where a Dom0 is not necessary

You can see the progress of our re-architecture in our latest release, Xen Project hypervisor 4.11. Also, the following summit presentations were relevant: here (Xen and automotive at Samsung) here (CPUFreq) and here (Real-time support).

These are just a few features and updates that make it easier for Xen to be used in embedded environments and market segments where safety certification is relevant. In addition, this will also significantly improve BoM and security in other market segments. On x86 we are also reducing code size, but this is significantly harder because of backward compatibility guarantees for x86 hardware and older operating systems.

Conclusion
The event was a great success with a lot of community and technical topics, like “How to Get Your Code Into Xen” and “The Art of Virtualizing Cache Maintenance.” Find the playlist for the full conference here. Additionally, our design sessions focused on architecture, embedded and safety, security, performance, and working practices and processes. You can find what was discussed, and next steps with these areas on our wiki.

If you want to stay abreast of where and when the Xen Project Developer and Design Summit will be held next year, follow us on Facebook and Twitter.

Xen Project 4.10.1 Available

I am pleased to announce the release of 4.10.1. Xen Project Maintenance releases are released in line with our Maintenance Release Policy. We recommend that all users of the 4.10 stable series update to the latest point release.

These releases are available from their git repositories

xenbits.xen.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.10 (tag RELEASE-4.10.1)

or from the XenProject download page

www.xenproject.org/downloads/xen-archives/xen-project-410-series/xen-4101.html

These releases contain many bug fixes and improvements. For a complete list of changes, please check the lists of changes on the download pages.

Announcing the Xen Project 4.11 RC and Test Day Schedules

On Tuesday, we created Xen 4.11 RC1 and will release a new release candidate every FRIDAY, until we declare a release candidate as the final candidate and cut the Xen 4.11 release. We will also hold a Test Day every TUESDAY for the release candidate that was released the week prior to the Test Day starting from RC1. Note that RC’s are announced on the following mailing lists: xen-announce, xen-devel and xen-users. This means we will have Test Days on April 24th, May 1st, 8th, 15th and 22nd. Your testing is still valuable on other days, so please feel free to send Test Reports as outlined below at any time.

Getting, Building and Installing a Release Candidate

Release candidates are available from our git repository at

git://xenbits.xenproject.org/xen.git (tag 4.11.0-<rc>)

where <rc> is rc1, rc2, rc3, etc. and as tarball from

https://downloads.xenproject.org/release/xen/4.11.0-<rc>/xen-4.11.0-<rc>.tar.gz
https://downloads.xenproject.org/release/xen/4.11.0-<rc>/xen-4.11.0-<rc>.tar.gz.sig

Detailed build and Install instructions can be found on the Test Day Wiki. Make sure you check the known issues section of the instructions before trying to download an RC.

Testing new Features, Test and Bug Reports

You can find Test Instructions for new features on our Test Day Wiki and instructions for general tests on Testing Xen. The following pages provide information on how to report successful tests and how to report bugs and issues.

Happy Testing!

The Xen Project is participating in 2018 Summer round of Outreachy

This is a quick reminder that the Xen Project is again participating in Outreachy (May 2018 to August 2018 Round). Please check the Outreachy application page for more information.

Outreach Program for Women has been helping women (cis and trans), trans men, and genderqueer people get involved in free and open source software worldwide. Note that the program has been extended and is now also open to people from other groups underrepresented in FOSS: specifically the program is open to residents and nationals of the United States of any gender who are Black/African American, Hispanic/Latin, American Indian, Alaska Native, Native Hawaiian, or Pacific Islander. Information on Eligibility and the application process can be found here.

Meet us at FOSDEM 2018

As in the past, the Xen Project will have a booth at Europe’s biggest open source conference FOSDEM (taking place February 3rd and 4th in Brussels, Belgium).

Where?

During FOSDEM community volunteers will man our booth, which is located in bulding K (level 1, group C).

fosdem-event-2018

Meet the Team!

You will have the opportunity to speak to some of our developers: Anthony Perard (Maintainer of various components and qemu developer, Citrix), Daniel Kiper (Xen and Grub developer, Oracle) – Daniel will be mostly on the Grub booth, Ian Jackson (Committer and maintainer of various components, Citrix), Lars Kurth (Community Manager, Citrix), Roger Pau Monné (Maintainer of various components most recently PVH, Citrix), Simon Kainzer (Project Lead of Unikraft, NEC) and Wim ten Have (Xen and libvirt developer, Oracle). We also have Julien Fontanet (CTO, Vates) and Olivier Lambert (CEO, Vates) from Xen Orchestra at our booth!

We also have a few talks in the Virtualisation Devroom.

Xen Project Spectre / Meltdown FAQ (Jan 22 Update)

This document has been updated since it was published! Updates are marked in blue.

On January 3rd, 2018, Google’s Project Zero announced several information leak vulnerabilities affecting all modern superscalar processors. Details can be found on their blog, and in the Xen Project Advisory 254. To help our users understand the impact and our next steps forward, we put together the following FAQ. We divided the FAQ into several sections to make it easier to consume:

  • General considerations affecting all 3 vulnerabilities
  • “Rogue Data Load” (aka SP3, “Variant 3”, Meltdown, CVE-2017-5754)
  • “Branch Target Injection” (aka SP2, “Variant 2”, Spectre CVE-2017-5715)
  • “Bounds-check bypass” (aka SP1, “Variant 1”, Spectre CVE-2017-5753)

The project has been developing patches in order of exploitability. Our initial focus was on fixes for Meltdown, then on fixes for Spectre Variant 2, and finally on Variant 1.

Generally in the context of Xen based systems, there are many different considerations that have gone into our strategy, such as

  • Can a guest (user or kernel space) attack the hypervisor using Meltdown or Spectre?
  • Can a guest (user or kernel space) attack another guest (user or kernel space) when running in a Xen VM?

Note that impact and mitigations are specific to CPU architectures (and in some cases models) and may also differ depending on virtualization mode. The below FAQ tries to lay this out clearly, but if you have any questions, please reply to this email thread.

Note that we will update or re-issue the FAQ on this blog as new information surfaces.

1) General Questions related to all 3 vulnerabilities

1.1) Is there any risk of privilege escalation?

Meltdown and Spectre are, by themselves, only information leaks. There is no suggestion that speculative execution can be used to modify memory or cause the system to do anything it might not have done already.

1.2) Where can I find more information?

We will update this blog post and Advisory 254 as new information becomes available. Updates will also be published on the xen-announce@ mailing list.

1.3) Where can I ask questions?

This blog post, has been posted in text form on the xen-devel@ mailing list. If you have questions or improvement suggestions, please reply to this email thread.

1.4) Where does development of mitigations happen?

As information related to Meltdown and Spectre is now public, development will continue in public on xen-devel@ and patches will be posted and attached to Advisory 254 as they become available.

2) SP3, “Variant 3”, Meltdown, CVE-2017-5754

2.1) Is Xen impacted by Meltdown (“Variant 3”)?

Only Intel processors are impacted by Meltdown (referred to as SP3 in Advisory 254). On Intel processors, only 64-bit PV mode guests can attack Xen. Guests running in 32-bit PV mode, HVM mode, and PVH mode cannot attack the hypervisor using SP3.

Note that in general, some ARM processors are impacted by Meltdown (see https://developer.arm.com/support/security-update): however these cannot be exploited on Xen.

Guest Type

Is a user space attack from a guest to Xen possible?

Is a kernel space attack from a guest to Xen possible?

Available Mitigations

32 bit PV

No

No

N/A

64 bit PV

Yes

Yes

Several with different trade-offs
See Question 2.2

HVM

No

No

N/A

PVH

No

No

N/A

ARM [1]

No

No

N/A

Notes:
[1] ARM’s security update refers to a subvariant of Meltdown called “Variant 3a”. The impact analysis of this variant is not yet fully complete, but we believe that no sensitive data can be leaked to exploit Xen.

2.2) Are there any patches available for Meltdown (“Variant 3”)?

The project has published five different mitigations with Advisory 254 following different mitigation strategies for Meltdown. Two strategies involve switching from PV guests to PVH or HVM guests. The others require application of patches as outlined in Advisory 254:

  • Vixen: The basic principle is to run PV guests (which can read all of host memory due to Meltdown) as HVM guests (which cannot read memory due to Meltdown) using a hypervisor shim.
  • Comet: The basic principle is to run PV guests (which can read all of host memory due to the hardware bugs) as PVH guests (which cannot read memory due to Meltdown) using a hypervisor shim.
  • PTI or Xen PTI stage-1: This solution implements Page Table Isolation (PTI) for Xen.

Each strategy has different trade-offs and will work well for some use-cases, but not others. A high-level comparison of the different trade-offs for each mitigation, including information about code and documentation can be found in Advisory 254 (under “SP3 MITIGATION OPTIONS SUMMARY TABLE FOR 64-bit X86 PV GUESTS”). Please make sure you carefully read this section and the README files in the advisory.

2.3) How are Xen Guests impacted by Meltdown (“Variant 3”)?

In 32-bit PV mode, HVM mode, and PVH mode, guest user spaces can attack guest kernels using SP3; so updating guest kernels is advisable. Interestingly, guest kernels running in 64-bit PV mode are not vulnerable to attack using SP3, but attacks on user and kernel spaces of other guests are possible.

Guest Type

Is a user space attack on the guest kernel possible (when running in a Xen VM)?

Is a user space attack on other guests possible (when running in a Xen VM)?

Is a kernel space attack on other guests possible (when running in a Xen VM)?

32 bit PV

Yes [1]

No

No

64 bit PV

No [2]

Yes [3]

Yes [3]

HVM

Yes [1]

No

No

PVH

Yes [1]

No

No

ARM

Yes [1]

No

No

Mitigations and notes:
[1] Can be mitigated by the Linux KPTI patch and similar patches for other operating systems
[2] Although, a direct user space attack on the kernel is not possible, user space can indirectly be exploited via [3]. When Vixen and Comet are deployed, all guest memory is mapped by the “shim,” which is itself vulnerable to Meltdown. The Xen PTI patches protect both the hypervisor and the guest kernel from attacks from the guest user (without need for additional guest kernel patches). Note that KPTI is automatically disabled when running in 64 bit PV guests: thus running Xen PTI together with KPTI should not have any adverse effects.
[3] Mitigated by stage-1 Xen PTI

2.4) What is the long-term plan for Meltdown (“Variant 3”)?

Longer term, we will merge Vixen with Comet and release in suitable Xen Point releases with the codename Rudolph. In addition, we will improve PTI. We will likely backport and release PTI in suitable Xen point releases.

Note that Vixen and Comet will not be released in Xen point releases, but only through Advisory 254.

2.5) Does Xen have any equivalent to Linux’s KPTI series?

Linux’s KPTI series is designed to address SP3 only. For Xen guests, only 64-bit PV guests are affected by SP3. We have released a PTI (sometimes called Xen PTI or XPTI) series, which we will continue to improve over the coming weeks.

3) SP2, “Variant 2”, Spectre, CVE-2017-5715

3.1) Is Xen impacted by Spectre (“Variant 2”)?

Both Intel and AMD CPUs are vulnerable to Spectre (both variants). Vulnerability of ARM processors to Spectre (both variants) varies by model and manufacturer.

Guest Type

Is a user space attack from a guest to Xen possible?

Is a kernel space attack from a guest to Xen possible?

Available Mitigations

x86

Yes

Yes

See Question 3.4.1

ARM 32 [1]

Yes

Yes

See Question 3.4.2

ARM 64 [1]

Yes

Yes

Mitigations and notes:
[1] ARM has information on affected models on the following website: https://developer.arm.com/support/security-update. According to Cavium Thunder X1 is not vulnerable to Spectre (both variants).

3.2) How are Xen Guests impacted by Spectre (“Variant 2”)?

Both Intel and AMD CPUs are vulnerable to Spectre (both variants). Vulnerability of ARM processors to Spectre (both variants) varies by model and manufacturer.

Guest Type

Is a user space attack on other user processes or the guest kernel within the same guest possible (when running in a Xen VM)?

Is a user space attack on other guests possible (when running in a Xen VM)?

Is a kernel space attack on other guests possible (when running in a Xen VM)?

x86

Yes [2]

Yes [3]

Yes [3]

ARM 32 [1]

Yes [2]

Yes [4]

Yes [4]

ARM 64 [1]

Yes [2]

Yes [5]

Yes [5]

Mitigations and notes:
[1] ARM has information on affected models on the following website: https://developer.arm.com/support/security-update. According to Cavium Thunder X1 is not vulnerable to Spectre (both variants).
[2] Mitigated by retpoline or firmware based Indirect Branch Control mitigations in guest operating systems (see here for Linux Kernel mitigations)
[3] Mitigated by “Intel and AMD CPUs” approach as outlined in question 3.4.1
[4] Mitigated by “Affected ARM CPUs” (64 bit) approach as outlined in question 3.4.2
[5] Mitigated by “Affected ARM CPUs” (32 bit) approach as outlined in question 3.4.2

3.3) Are mitigations for Spectre possible (“Variant 2”)?

SP2 can be mitigated in two ways, both of which essentially prevent speculative execution of indirect branches. The first is to flush the branch prediction logic on entry into the hypervisor. This requires microcode updates, which Intel and AMD are in the process of preparing, as well as patches to the hypervisor which are also in process and should be available soon. On ARM, firmware updates are required (see here).

The second is to do indirect jumps in a way that is not subject to speculative execution (this approach is called Retpoline). This requires the hypervisor to be recompiled with a compiler that contains special new features. These new compiler features are also in the process of being prepared for both GCC (see here and here) and clang, and should be available soon.

3.4) Are there any patches available for Spectre (“Variant 2”)?

3.4.1) Intel and AMD CPUs:

We have developed prototype patches for x86 CPUs. These patches depend on firmware updates. Our prototype patches were developed against pre-released versions of MSR specifications and were modified to comply with published MSR specifications (see here). This may require changes to our patches. There have also been reports of issues with some published firmware updates (see here and here) leading to frequent reboots of systems where these have been deployed. We are currently evaluating the situation to verify whether Xen based systems with mitigations are affected.

Variant 2 mitigations have been published via Advisory 254. More information on ongoing development can be found on relevant xen-devel@ discussions which are linked to from here.

3.4.2) Affected ARM CPUs:

A framework to mitigate Spectre Variant 2 has been developed (for 64 bit and 32 bit). CPU vendors, will be able to add support for specific CPUs to the framework.

The framework and vendor specific mitigations has been published via Advisory 254 (including support for ARM CPUs). More information on ongoing development can be found on relevant xen-devel@ discussions which are linked to from here.

4) SP1, “Variant 1”, Spectre, CVE-2017-5753

4.1) Is Xen impacted by Spectre (“Variant 1”)?

Both Intel and AMD CPUs are vulnerable to Spectre (both variants). Vulnerability of ARM processors to Spectre (both variants) varies by model and manufacturer.

Guest Type

Is a user space attack from a guest to Xen possible?

Is a kernel space attack from a guest to Xen possible?

Available Mitigations

x86

Yes

Yes

See Question 4.3

ARM [1]

Yes

Yes

Mitigations and notes:
[1] ARM has information on affected models on the following website: https://developer.arm.com/support/security-update. According to Cavium Thunder X1 is not vulnerable to Spectre (both variants).

4.2) How are Xen Guests impacted by Spectre (“Variant 1”)?

Both Intel and AMD CPUs are vulnerable to Spectre (both variants). Vulnerability of ARM processors to Spectre (both variants) varies by model and manufacturer.

Guest Type

Is a user space attack on other user processes or the guest kernel within the same guest possible (when running in a Xen VM)?

Is a user space attack on other guests possible (when running in a Xen VM)?

Is a kernel space attack on other guests possible (when running in a Xen VM)?

x86

Yes [2]

Yes [3]

Yes [3]

ARM [1]

Yes [2]

Yes [3]

Yes [3]

Mitigations and notes:
[1] ARM has information on affected models on the following website: https://developer.arm.com/support/security-update. According to Cavium Thunder X1 is not vulnerable to Spectre (both variants).
[2] Please refer to guest operating specific mitigations (see here for Linux Kernel mitigations)
[3] See question 4.3

4.3) Are mitigations for Spectre possible (“Variant 1”)?

Spectre Variant 1 is much more difficult to mitigate. We have some ideas we’re exploring, but they’re still at the design stage at this point.