Xen Project Community Spotlight: DornerWorks

The Xen Project is comprised of a diverse set of member companies and contributors that are committed to the growth and success of the Xen Project Hypervisor. The Xen Project Hypervisor is a staple technology for server and cloud vendors, and is gaining traction in the embedded, security and automotive space. This blog series highlights the companies contributing to the changes and growth being made to the Xen Project, and how the Xen Project technology bolsters their business.

contemporary-1850469_1920

Name: Robert VanVossen
Title: Embedded Engineer
Company: DornerWorks

When did you join the Xen Project and why/how is your organizations involved?DornerWorks has been involved with Xen Project since 2011 when we submitted the ARINC653 scheduler to the mainline. Through a Small Business Innovation Research (SBIR) contract from the US NAVY, we worked to develop some ARINC653 extensions to the Xen Project Hypervisor. This led to DornerWorks gaining expertise in the Xen Project Hypervisor and we combined this expertise with our knowledge of the embedded world to help our customers design the embedded virtualization solutions they need. This formed the basis for our Virtuosity product (a hypervisor distribution based on the Xen Project) and design services around embedded hypervisors.

DornerWorks still tries to propagate changes and bug fixes to the Xen Project Mainline whenever we can as we take great pride in being a part of this community. We want to help improve both the technology and the community through our work.

How does your involvement benefit your company?
A hypervisor is a complex piece of technology and DornerWorks is a small company.  By colloborating with the larger Xen community, DornerWorks is able to provide a competitive embedded virtualization solution without needing to become an expert at all the technology areas encapsulated in Xen Project technologies.

As a primarily services based company, the Xen Project community also provides us with an indirect marketing platform. The opportunity to publicly showcase our contributions and present on technical topics at Xen Project conferences allows us to share our expertise with the community while getting our name in front of potential customers.

The Xen Project community has also been instrumental in encouraging Xen’s use in embedded platforms, which while different from Xen’s original cloud based goals, is an area poised for growth in virtualization implementations.

How does the Xen Project’s technology help your business?
Xen Project technologies provide a basis for services that we provide to our customers. Through the DornerWorks Virtuosity distribution (http://dornerworks.com/xen/virtuosity), we give customers the means to get started quickly prototyping with Xen on embedded systems. From there, we provide services to refine their solution or develop new components around Xen that they may need, such as new guest OS, new PV drivers, etc. We also provide a Quick Start Package (http://dornerworks.com/xen/xen-quick-start) to help train others on Xen, virtualization, and specific platforms. This is a good option for both those that are just getting their feet wet and those that want to know all the nitty-gritty details.

What are some of the major changes you see with virtualization?
We see virtualization becoming more and more of a necessity in the embedded world. As the complexity of processors increases, the difficulty of utilizing them increases. Processors, like the Zynq UltraScale+ MPSoC, that have a Quad-Core ARM Cortex-A53, a Dual-Core ARM Cortex-R5, and an FPGA in a single chip, can be difficult to manage. Virtualization provides a means to isolate out various pieces in a more manageable and effective way. Not only does the Xen Project Hypervisor help manage complexity, but it also can reduce size, weight, and power (SWaP), provide redundancy, address obsolescence of legacy systems, and more.

However, while the temptation is to use virtualization to create a single integrated platform for all computation, this approach could create a single point of failure unless it is mitigated by system wide redundancies. In these applications, Xen Project technologies can be used to provide an embedded “cloud,” which provides the reliability required by the application with a large measure of integration. This approach is both familiar and different in embedded applications, which frequently use both hardware and software to provide both isolation and redundancy, but have traditionally leaned more on hardware based solutions.

What advice would you give someone considering joining the Xen Project?
Just jump in and get involved. Go to the conferences, meet people, submit patches, review patches, ask questions, and enjoy yourself. It is a great community that is friendly, open, and has a lot of people with similar goals. They want to help each other and improve the technologies we are all utilizing. I have personally had a blast at the Developer’s Summit and look forward to going to more.

What excites you most about the future of Xen?
I am excited to see hardware become more virtualization friendly. When Xen can utilize these features, the overhead added to the system can be decreased even further than it already has been. This will help make the Xen Project Hypervisor an even more attractive solution in the embedded space.

Embedded hypervisors have been around for a long time, but with the increasingly complex SoCs being produced by chip vendors and the industry drive towards system integration, the number of deployed hypervisor based embedded systems continues to increase. While it has taken longer than we thought when we first joined the Xen Project community, we can see the fruits of these efforts starting to pay off.  We are excited to be a part of the many Xen Project contributors putting Xen in systems quite different from the cloud, utilizing the same underlying technologies in order to provide the security and reliability we have become accustomed to in cloud applications to embedded ones.

Additionally when we first started working with the Xen Project there was not much talk about the safety certification of Xen, but with the increasing interest of the automotive industry in hypervisors, we are seeing a lot of discussion and progress on this front. There is still a long way to go, but at least the will is currently there.

*If you want to stay in the know around embedded virtualization and Xen, sign up for DornerWork’s weekly newsletter here

Xen Project Contributor Spotlight: Kevin Tian

The Xen Project is comprised of a diverse set of member companies and contributors that are committed to the growth and success of the Xen Project Hypervisor. The Xen Project Hypervisor is a staple technology for server and cloud vendors, and is gaining traction in the embedded, security and automotive space. This blog series highlights the companies contributing to the changes and growth being made to the Xen Project and how the Xen Project technology bolsters their business.

contemporary-1850469_1920

Name: Kevin Tian
Title: Principal Engineer of Open Source Technology Center
Company: Intel

When did you join the Xen Project and why/how is your organizations involved?
My journey with Xen Project has been ~13 years now (since 2005), with a focus on hardware-assisted virtualization using Intel® Virtualization Technology (Intel® VT). I’m acting as the maintainer for VT-x/VT-d sub-system in the Xen Project community. The Xen Project is the first open source virtualization project embracing Intel® VT and is a leading community in demonstrating new hardware virtualization features.

How does your involvement benefit your company?
Working with open source communities can definitely bring great value to the whole ecosystem around new technologies, which Intel debuts every year. For example, being the pioneer on Intel® VT, the success in the Xen Project accelerated the market transition from software-based virtualization (binary translation, para-virtualization, etc.) to hardware-assisted virtualization (HVM, PVH, etc.). Hardware-assisted virtualization helps with reduced maintenance overhead, full guest OS compatibility, and better performance.

How does the Xen Project’s technology help your business?
The ecosystem built around the Xen Project is definitely helpful in generating demand of Intel servers (with Intel® VT).

What are some of the major changes you see with virtualization and the transition to cloud native computing?
While virtualization technology has become the fundamental building block in the Cloud, there is still a major gap regarding I/O capabilities when comparing virtualized environment to bare metal. Although network and storage virtualization has been in place for years, efficient virtualization and sharing of new booming accelerators (GPU, NVMe, FPGA, QAT, etc.) are still not widely available. The ceiling of what cloud-native computing can achieve could be severely limited, if disconnected from powerful accelerators existing in the physical server.

What advice would you give someone considering joining the Xen Project?
The Xen Project is possibly one of the most successful open source virtualization projects in the world. The mature community and rich features accumulated in the decade plus the project has been in existence has provided a strong foundation to save you time either in developing a value-add business or exploiting new virtualization research.

What excites you most about the future of Xen?
I’m excited by the fact that the Xen Project keeps embracing new innovations, e.g. PVH, XenGT, etc., and penetrating new markets.

The Xen Project is participating in 2018 Summer round of Outreachy

This is a quick reminder that the Xen Project is again participating in Outreachy (May 2018 to August 2018 Round). Please check the Outreachy application page for more information.

Outreach Program for Women has been helping women (cis and trans), trans men, and genderqueer people get involved in free and open source software worldwide. Note that the program has been extended and is now also open to people from other groups underrepresented in FOSS: specifically the program is open to residents and nationals of the United States of any gender who are Black/African American, Hispanic/Latin, American Indian, Alaska Native, Native Hawaiian, or Pacific Islander. Information on Eligibility and the application process can be found here.

Meet us at FOSDEM 2018

As in the past, the Xen Project will have a booth at Europe’s biggest open source conference FOSDEM (taking place February 3rd and 4th in Brussels, Belgium).

Where?

During FOSDEM community volunteers will man our booth, which is located in bulding K (level 1, group C).

fosdem-event-2018

Meet the Team!

You will have the opportunity to speak to some of our developers: Anthony Perard (Maintainer of various components and qemu developer, Citrix), Daniel Kiper (Xen and Grub developer, Oracle) – Daniel will be mostly on the Grub booth, Ian Jackson (Committer and maintainer of various components, Citrix), Lars Kurth (Community Manager, Citrix), Roger Pau Monné (Maintainer of various components most recently PVH, Citrix), Simon Kainzer (Project Lead of Unikraft, NEC) and Wim ten Have (Xen and libvirt developer, Oracle). We also have Julien Fontanet (CTO, Vates) and Olivier Lambert (CEO, Vates) from Xen Orchestra at our booth!

We also have a few talks in the Virtualisation Devroom.

Xen Project 4.8.3 is available

I am pleased to announce the release of Xen 4.8.3. Xen Project Maintenance releases are released in line with our Maintenance Release Policy.

We recommend that all users of the 4.8 stable series update to the latest point release. The release is available from its git repository

xenbits.xenproject.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.8 (tag RELEASE-4.8.3)

or from the Xen Project download page

www.xenproject.org/downloads/xen-archives/xen-48-series/xen-483.html

These releases contain many bug fixes and improvements. You can find a complete list of changes and the release notes on the download page.

Xen Project Spectre / Meltdown FAQ (Jan 22 Update)

This document has been updated since it was published! Updates are marked in blue.

On January 3rd, 2018, Google’s Project Zero announced several information leak vulnerabilities affecting all modern superscalar processors. Details can be found on their blog, and in the Xen Project Advisory 254. To help our users understand the impact and our next steps forward, we put together the following FAQ. We divided the FAQ into several sections to make it easier to consume:

  • General considerations affecting all 3 vulnerabilities
  • “Rogue Data Load” (aka SP3, “Variant 3”, Meltdown, CVE-2017-5754)
  • “Branch Target Injection” (aka SP2, “Variant 2”, Spectre CVE-2017-5715)
  • “Bounds-check bypass” (aka SP1, “Variant 1”, Spectre CVE-2017-5753)

The project has been developing patches in order of exploitability. Our initial focus was on fixes for Meltdown, then on fixes for Spectre Variant 2, and finally on Variant 1.

Generally in the context of Xen based systems, there are many different considerations that have gone into our strategy, such as

  • Can a guest (user or kernel space) attack the hypervisor using Meltdown or Spectre?
  • Can a guest (user or kernel space) attack another guest (user or kernel space) when running in a Xen VM?

Note that impact and mitigations are specific to CPU architectures (and in some cases models) and may also differ depending on virtualization mode. The below FAQ tries to lay this out clearly, but if you have any questions, please reply to this email thread.

Note that we will update or re-issue the FAQ on this blog as new information surfaces.

1) General Questions related to all 3 vulnerabilities

1.1) Is there any risk of privilege escalation?

Meltdown and Spectre are, by themselves, only information leaks. There is no suggestion that speculative execution can be used to modify memory or cause the system to do anything it might not have done already.

1.2) Where can I find more information?

We will update this blog post and Advisory 254 as new information becomes available. Updates will also be published on the xen-announce@ mailing list.

1.3) Where can I ask questions?

This blog post, has been posted in text form on the xen-devel@ mailing list. If you have questions or improvement suggestions, please reply to this email thread.

1.4) Where does development of mitigations happen?

As information related to Meltdown and Spectre is now public, development will continue in public on xen-devel@ and patches will be posted and attached to Advisory 254 as they become available.

2) SP3, “Variant 3”, Meltdown, CVE-2017-5754

2.1) Is Xen impacted by Meltdown (“Variant 3”)?

Only Intel processors are impacted by Meltdown (referred to as SP3 in Advisory 254). On Intel processors, only 64-bit PV mode guests can attack Xen. Guests running in 32-bit PV mode, HVM mode, and PVH mode cannot attack the hypervisor using SP3.

Note that in general, some ARM processors are impacted by Meltdown (see https://developer.arm.com/support/security-update): however these cannot be exploited on Xen.

Guest Type

Is a user space attack from a guest to Xen possible?

Is a kernel space attack from a guest to Xen possible?

Available Mitigations

32 bit PV

No

No

N/A

64 bit PV

Yes

Yes

Several with different trade-offs
See Question 2.2

HVM

No

No

N/A

PVH

No

No

N/A

ARM [1]

No

No

N/A

Notes:
[1] ARM’s security update refers to a subvariant of Meltdown called “Variant 3a”. The impact analysis of this variant is not yet fully complete, but we believe that no sensitive data can be leaked to exploit Xen.

2.2) Are there any patches available for Meltdown (“Variant 3”)?

The project has published five different mitigations with Advisory 254 following different mitigation strategies for Meltdown. Two strategies involve switching from PV guests to PVH or HVM guests. The others require application of patches as outlined in Advisory 254:

  • Vixen: The basic principle is to run PV guests (which can read all of host memory due to Meltdown) as HVM guests (which cannot read memory due to Meltdown) using a hypervisor shim.
  • Comet: The basic principle is to run PV guests (which can read all of host memory due to the hardware bugs) as PVH guests (which cannot read memory due to Meltdown) using a hypervisor shim.
  • PTI or Xen PTI stage-1: This solution implements Page Table Isolation (PTI) for Xen.

Each strategy has different trade-offs and will work well for some use-cases, but not others. A high-level comparison of the different trade-offs for each mitigation, including information about code and documentation can be found in Advisory 254 (under “SP3 MITIGATION OPTIONS SUMMARY TABLE FOR 64-bit X86 PV GUESTS”). Please make sure you carefully read this section and the README files in the advisory.

2.3) How are Xen Guests impacted by Meltdown (“Variant 3”)?

In 32-bit PV mode, HVM mode, and PVH mode, guest user spaces can attack guest kernels using SP3; so updating guest kernels is advisable. Interestingly, guest kernels running in 64-bit PV mode are not vulnerable to attack using SP3, but attacks on user and kernel spaces of other guests are possible.

Guest Type

Is a user space attack on the guest kernel possible (when running in a Xen VM)?

Is a user space attack on other guests possible (when running in a Xen VM)?

Is a kernel space attack on other guests possible (when running in a Xen VM)?

32 bit PV

Yes [1]

No

No

64 bit PV

No [2]

Yes [3]

Yes [3]

HVM

Yes [1]

No

No

PVH

Yes [1]

No

No

ARM

Yes [1]

No

No

Mitigations and notes:
[1] Can be mitigated by the Linux KPTI patch and similar patches for other operating systems
[2] Although, a direct user space attack on the kernel is not possible, user space can indirectly be exploited via [3]. When Vixen and Comet are deployed, all guest memory is mapped by the “shim,” which is itself vulnerable to Meltdown. The Xen PTI patches protect both the hypervisor and the guest kernel from attacks from the guest user (without need for additional guest kernel patches). Note that KPTI is automatically disabled when running in 64 bit PV guests: thus running Xen PTI together with KPTI should not have any adverse effects.
[3] Mitigated by stage-1 Xen PTI

2.4) What is the long-term plan for Meltdown (“Variant 3”)?

Longer term, we will merge Vixen with Comet and release in suitable Xen Point releases with the codename Rudolph. In addition, we will improve PTI. We will likely backport and release PTI in suitable Xen point releases.

Note that Vixen and Comet will not be released in Xen point releases, but only through Advisory 254.

2.5) Does Xen have any equivalent to Linux’s KPTI series?

Linux’s KPTI series is designed to address SP3 only. For Xen guests, only 64-bit PV guests are affected by SP3. We have released a PTI (sometimes called Xen PTI or XPTI) series, which we will continue to improve over the coming weeks.

3) SP2, “Variant 2”, Spectre, CVE-2017-5715

3.1) Is Xen impacted by Spectre (“Variant 2”)?

Both Intel and AMD CPUs are vulnerable to Spectre (both variants). Vulnerability of ARM processors to Spectre (both variants) varies by model and manufacturer.

Guest Type

Is a user space attack from a guest to Xen possible?

Is a kernel space attack from a guest to Xen possible?

Available Mitigations

x86

Yes

Yes

See Question 3.4.1

ARM 32 [1]

Yes

Yes

See Question 3.4.2

ARM 64 [1]

Yes

Yes

Mitigations and notes:
[1] ARM has information on affected models on the following website: https://developer.arm.com/support/security-update. According to Cavium Thunder X1 is not vulnerable to Spectre (both variants).

3.2) How are Xen Guests impacted by Spectre (“Variant 2”)?

Both Intel and AMD CPUs are vulnerable to Spectre (both variants). Vulnerability of ARM processors to Spectre (both variants) varies by model and manufacturer.

Guest Type

Is a user space attack on other user processes or the guest kernel within the same guest possible (when running in a Xen VM)?

Is a user space attack on other guests possible (when running in a Xen VM)?

Is a kernel space attack on other guests possible (when running in a Xen VM)?

x86

Yes [2]

Yes [3]

Yes [3]

ARM 32 [1]

Yes [2]

Yes [4]

Yes [4]

ARM 64 [1]

Yes [2]

Yes [5]

Yes [5]

Mitigations and notes:
[1] ARM has information on affected models on the following website: https://developer.arm.com/support/security-update. According to Cavium Thunder X1 is not vulnerable to Spectre (both variants).
[2] Mitigated by retpoline or firmware based Indirect Branch Control mitigations in guest operating systems (see here for Linux Kernel mitigations)
[3] Mitigated by “Intel and AMD CPUs” approach as outlined in question 3.4.1
[4] Mitigated by “Affected ARM CPUs” (64 bit) approach as outlined in question 3.4.2
[5] Mitigated by “Affected ARM CPUs” (32 bit) approach as outlined in question 3.4.2

3.3) Are mitigations for Spectre possible (“Variant 2”)?

SP2 can be mitigated in two ways, both of which essentially prevent speculative execution of indirect branches. The first is to flush the branch prediction logic on entry into the hypervisor. This requires microcode updates, which Intel and AMD are in the process of preparing, as well as patches to the hypervisor which are also in process and should be available soon. On ARM, firmware updates are required (see here).

The second is to do indirect jumps in a way that is not subject to speculative execution (this approach is called Retpoline). This requires the hypervisor to be recompiled with a compiler that contains special new features. These new compiler features are also in the process of being prepared for both GCC (see here and here) and clang, and should be available soon.

3.4) Are there any patches available for Spectre (“Variant 2”)?

3.4.1) Intel and AMD CPUs:

We have developed prototype patches for x86 CPUs. These patches depend on firmware updates. Our prototype patches were developed against pre-released versions of MSR specifications and were modified to comply with published MSR specifications (see here). This may require changes to our patches. There have also been reports of issues with some published firmware updates (see here and here) leading to frequent reboots of systems where these have been deployed. We are currently evaluating the situation to verify whether Xen based systems with mitigations are affected.

Variant 2 mitigations have been published via Advisory 254. More information on ongoing development can be found on relevant xen-devel@ discussions which are linked to from here.

3.4.2) Affected ARM CPUs:

A framework to mitigate Spectre Variant 2 has been developed (for 64 bit and 32 bit). CPU vendors, will be able to add support for specific CPUs to the framework.

The framework and vendor specific mitigations has been published via Advisory 254 (including support for ARM CPUs). More information on ongoing development can be found on relevant xen-devel@ discussions which are linked to from here.

4) SP1, “Variant 1”, Spectre, CVE-2017-5753

4.1) Is Xen impacted by Spectre (“Variant 1”)?

Both Intel and AMD CPUs are vulnerable to Spectre (both variants). Vulnerability of ARM processors to Spectre (both variants) varies by model and manufacturer.

Guest Type

Is a user space attack from a guest to Xen possible?

Is a kernel space attack from a guest to Xen possible?

Available Mitigations

x86

Yes

Yes

See Question 4.3

ARM [1]

Yes

Yes

Mitigations and notes:
[1] ARM has information on affected models on the following website: https://developer.arm.com/support/security-update. According to Cavium Thunder X1 is not vulnerable to Spectre (both variants).

4.2) How are Xen Guests impacted by Spectre (“Variant 1”)?

Both Intel and AMD CPUs are vulnerable to Spectre (both variants). Vulnerability of ARM processors to Spectre (both variants) varies by model and manufacturer.

Guest Type

Is a user space attack on other user processes or the guest kernel within the same guest possible (when running in a Xen VM)?

Is a user space attack on other guests possible (when running in a Xen VM)?

Is a kernel space attack on other guests possible (when running in a Xen VM)?

x86

Yes [2]

Yes [3]

Yes [3]

ARM [1]

Yes [2]

Yes [3]

Yes [3]

Mitigations and notes:
[1] ARM has information on affected models on the following website: https://developer.arm.com/support/security-update. According to Cavium Thunder X1 is not vulnerable to Spectre (both variants).
[2] Please refer to guest operating specific mitigations (see here for Linux Kernel mitigations)
[3] See question 4.3

4.3) Are mitigations for Spectre possible (“Variant 1”)?

Spectre Variant 1 is much more difficult to mitigate. We have some ideas we’re exploring, but they’re still at the design stage at this point.