Q&A with GlobalLogic on the Xen Project and Automotive Virtualization

The Xen Project is commonly used in embedded scenarios due to its security features, light-weight architecture and open source community. These core attributes are now making it more pervasive in the automotive industry, which has similar demands to the embedded industry, especially when it comes to security requirements.

To better understand how the Xen Project is used in the automotive space, we sat down with the folks at GlobalLogic to discuss updates on its Nautilus platform, which uses the Xen Project hypervisor; why they originally chose Xen; how hypervisors generally work in the automotive space; and the company’s upcoming plans with automotive virtualization.

Last year when we talked to GlobalLogic, you mentioned that GPU Virtualization was the next phase of automotive innovation. Where are you at in terms of implementing GPU Virtualization?

We have successfully implemented our Nautilus platform’s GPU virtualization feature for several Tier 1 automotive vendors (located in Japan, the US, and Europe). This was a big win for us and we learned a lot along the way and experienced some major benefits. Mainly, GPU virtualization has eliminated almost all performance degradation during the rendering of heavy 3D graphics scenes, allowing us to create a new level of IVI systems.

Why is the hypervisor important for automotive virtualization and GPU Virtualization in general? Why is Xen Project the hypervisor of choice for you within this space?

The hypervisor allows a significant decrease to the cost of automotive production and reduces the cost of BOM because the functions that were previously executed on different CPUs can be run on separate VMs. At the same time, GPU virtualization is beneficial in the process of 2D/3D graphics rendering. Therefore, the use of hypervisor enables building systems that perform better than their more expensive completely-hardware analogues.

Moreover, there are less processors per board, which leads to higher fail-safety. Essentially, a virtual system divided into a number of small subsystems is cheaper to maintain.

At the dawn of our project, GlobalLogic engineers considered various hypervisors, and finally decided that Xen Project was the most suitable solution because it is open source and has a rich history of application in various fields. Using the Xen Project, lets us concentrate on specific vehicle-related challenges instead of reinventing a virtualization solution.

What are the top three benefits you get from using the Xen hypervisor?

The first benefit that we have experienced is the decreased time to market for the manufacturers. Secondly, our customers get demos for free – if we used a proprietary product, we couldn’t afford this. Finally, it is great to experience the constant support of the global community and the community-driven approach to vulnerability detecting and fixing that we get with the Xen Project.

Were there any challenges with implementing Xen? How did you overcome these challenges?

The main challenges that we had with Xen and GPU virtualization was related to the different based ARM platforms. To overcome this, we developed a bench of drivers and extended the environment around them.

What are the next stages of growth for with automotive virtualization? Any trends that we should watch out for?

GlobalLogic is actively working on the commercialization of the Nautilus platform. We are expanding the GPU feature to a network of customers and vehicle models. At the same time, we are expanding the functionality of virtualization in areas like self-driving, advanced driver assistance systems (ADAS), connected services, safety, etc.

Xen Hypervisor to be Rewritten

The hypervisor team has come to the conclusion that using the C programming language, which is 45 years old as of writing, is not a good idea for the long term success of the project.

C, without doubt, is ridden with quirks and undefined behaviours. Even the most experienced developers find this collection of powerful footguns difficult to use. We’re glad that the development of programming languages in the last decade has given us an abundance of better choices.

After a heated debate among committers, we’ve settled on picking two of the most popular languages on HackerNews to rewrite the Xen hypervisor project. Our winners are Rust and JavaScript.

Rust, although not old enough to drink, has attracted significant attention in recent years. The hypervisor maintainers have acquainted themselves with the ownership system, borrow checker, lifetimes and cargo build system. We will soon start rewriting the X86 exception handler entry point, which has been a major source of security bugs in the past, and looks like an easy starting point for the conversion to Rust.

JavaScript has been a corner stone of web development since early 2000. With the advancement of React Native and Electron, plus the exemplary success of Atom and Visual Studio Code editors, it now makes sense to start rebuilding the Xen hypervisor toolstack in JavaScript. We’re confident that Node.js would be of great help when it comes to performance. And we believe Node.js and the current libxenlight event model is a match made in heaven.

Due to the improved ergonomics of the two programming languages, we expect developer efficiency to be boosted by factor of 10. We’re also quite optimistic that we can tap into the large talent pool of Rust and JavaScript developers and get significant help from them. We expect the rewrite to be finished and released within the year – by April 2018.

For those who want a more solid, tried and true technology, we are open to the idea of toolstack middleware being written in PHP and frontend JavaScript. But since maintainers are too busy playing with their new shiny toys, those who want PHP middleware will have to step up and help.

Stay put and get ready to embrace the most secure and easy to use Xen hypervisor ever, on April 1st 2018!

Note that this article was an April fools joke and was entirely made up.

Xen on ARM interrupt latency

Xen on ARM is becoming more and more widespread in embedded environments. In these contexts, Xen is employed as a single solution to partition the system into multiple domains, fully isolated from each other, and with different levels of trust.

Every embedded scenario is different, but many require real-time guarantees. It comes down to interrupt latency: the hypervisor has to be able to deliver interrupts to virtual machines within a very small timeframe. The maximum tolerable lag changes on a case by case basis, but it should be in the realm of nanoseconds and microseconds, not milliseconds.

Xen on ARM meets these requirements in a few different ways. Firstly, Xen comes with a flexible scheduler architecture. Xen includes a set of virtual machine schedulers, including RTDS, a soft real-time scheduler, and ARINC653, a hard real-time scheduler. Users can pick the ones that perform best for their use-cases. However, if they really care about latency, the best option is to have no schedulers at all and use a static assignment for virtual cpus to physical cpus instead. There are no automatic ways to do that today, but it is quite easy to achieve with the vcpu-pin command:

Usage: xl vcpu-pin [domain-id] [vcpu-id] [pcpu-id]

For example, in a system with four physical cpus and two domains with two vcpus each, a user can get a static configuration with the following commands:

xl vcpu-pin 0 0 0
xl vcpu-pin 0 1 1
xl vcpu-pin 1 0 2
xl vcpu-pin 1 1 3

As a result, all vcpus are pinned to different physical cpus. In such a static configuration, the latency overhead introduced by Xen is minimal. Xen always configures interrupts to target the cpu that is running the virtual cpu that should receive the interrupt. Thus, the overhead is down to just the time that it takes to execute the code in Xen to handle the physical interrupt and inject the corresponding virtual interrupt to the vcpu.

For my measurements, I used a Xilinx Zynq Ultrascale+ MPSoC, an excellent board with four Cortex A53 cores and a GICv2 interrupt controller. I installed Xen 4.9 unstable (changeset 55a04feaa1f8ab6ef7d723fbb1d39c6b96ad184a) and Linux 4.6.0 as Dom0. I ran tbm as a guest, which is a tiny baremetal application that programs timer events in the future, then, after receiving them, checks the current time again to measure the latency. tbm uses the virtual timer for measurements, however, the virtual timer interrupt is handled differently compared to any other interrupts in Xen. Thus, to make the results more generally applicable, I modified tbm to use the physical timer interrupt instead. I also modified Xen to forward physical timer interrupts to guests.

Keeping in mind that the native interrupt latency is about 300ns on this board, these are the results on Xen in nanoseconds:

AVG  MIN  MAX  WARM_MAX
4850 4810 7030 4980

AVG is the average latency, MIN is the minimum, MAX is the maximum and WARM_MAX is the maximum latency observed after discarding the first few interrupts to warm the caches. For real-time considerations, the number to keep in mind is WARM_MAX, which is 5000ns (when using static vcpu assignments).

This excellent result is small enough for most use cases, including piloting a flying drone. However, it can be further improved by using the new vwfi Xen command line option. Specifically, when vcpus are statically assigned to physical cpus using vcpu-pin, it makes sense to pass vwfi=native to Xen: it tells the hypervisor not to trap wfi and wfe commands, which are ARM instructions for sleeping. If no other vcpus can ever be scheduled on a given physical cpu, then we might as well let the guest put the cpu to sleep. Passing vwfi=native, the results are:

AVG  MIN  MAX  WARM_MAX
1850 1680 2650 1950

With this configuration, the latency is only 2 microseconds, which is extremely close to the hardware limits, and should be small enough for the vast majority of use cases. vwfi was introduced recently, but it has been backported to all the Xen stable trees.

In addition to vcpu pinning and vwfi, the third key parameter to reduce interrupt latency is unexpectedly simple: the DEBUG kconfig option in Xen. DEBUG is enabled by default in all cases except for releases. It adds many useful debug messages and checks, at the cost of increased latency. Make sure to disable it in production and when doing measurements.

Now Accepting Submissions for Xen Project Developer and Design Summit 2017

screen-shot-2017-03-14-at-17-02-17 
We’re excited to announce that registration and the call for proposals is open for Xen Project Developer and Design Summit 2017, which will be held in Budapest, Hungary from July 11-13, 2017. The Xen Project Developer and Design Summit combines the formats of Xen Project Developer Summits with Xen Project Hackathons, and brings together the Xen Project’s community of developers and power users.

Submit a Talk

Do you have an interesting use case around Xen Project technology or best practices around the community? There’s a wide variety of topics we are looking for, including security, embedded environments, network function virtualization (NFV), and more. You can find all the suggested topics for presentations and panels here (make sure you select the Topics tab).

Several formats are being accepted for speaking proposals, including:

  • Presentations and Panels
  • Interactive design and problem solving sessions. These sessions can be submitted as part of the CFP, but we will reserve a number of design sessions to be allocated during the event. Proposers of design sessions are expected to host and moderate design sessions following the format we have used at Xen Project Hackathons. If you have not participated in these in the past, check out past event reports from 2016, 2015 and 2013.

Never talked at a conference before? Don’t worry! We encourage new speakers to submit for our events and have plenty of resources to help you prepare for your presentation.

Here are some dates to remember for submissions and in general:

  • CFP Close: April 14, 2017 (correction: was extended to April 21)
  • CFP Notifications: May 5, 2017
  • Schedule Announced: May 16, 2017
  • Event: July 11-13, 2017

Registration

Come join us for this event, and if you register by May 19, you’ll get an early bird discount :) Travel stipends are available for students or individuals that are not associated with a company. If you have any questions, please send a note to community.manager@xenproject.org.

Xen Project Maintenance Releases Available (Versions 4.6.5 and 4.7.2)

I am pleased to announce the release of Xen 4.6.5 and 4.7.2. Xen Project Maintenance releases are released in line with our Maintenance Release Policy. We recommend that all users of the 4.6 and 4.7 stable series update to the latest point release.

Xen 4.6.5

Xen 4.6.5 is available immediately from its git repository http://xenbits.xen.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.6
(tag RELEASE-4.6.5) or from the Xen Project download page http://www.xenproject.org/downloads/xen-archives/supported-xen-46-series/xen-465.html

Xen 4.7.2

Xen 4.7.2 is available immediately from its git repository http://xenbits.xen.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.7
(tag RELEASE-4.7.2) or from the Xen Project download page http://www.xenproject.org/downloads/xen-archives/supported-xen-47-series/xen-472.html

These releases contain many bug fixes and improvements. For a complete list of changes, please check the lists of changes on the download pages.

Xen Project Participates in Google Summer of Code and Outreachy

This is a quick announcement that the Xen Project is again participating in Google Summer of Code (GSoC), a program that awards three-month paid stipends to University students to work on open source projects, with the goal to get open source experience. The Xen Project will also again participate in Outreachy, which is an internship program that organises three-month paid internships with free and open-source software projects for people who are typically underrepresented in the technology industry. Outreachy has been helping women (cis and trans), trans men, genderqueer people and people from other discriminated backgrounds get involved in free and open source software for several years. The Xen Project is proud that it has participated continually in Outreachy (and its predecessor OPW) for 4 years.

I want to participate, how do I get started?

If you are not at all familiar with programs such as GSoC and Outreachy, have a quick look at our introduction. In a nutshell, both programs go through several stages:

  • Check eligibility requirements.
  • Now until application period: Preparation by working on small tasks (also called micro-tasks) within our community to identify a suitable project and to familiarise yourself with the technology.
  • Application Period (aka paperwork): For GSoC, the application system is open from March 20 to Apr 3, 2017; however you should work on micro-tasks before and prepare your application together with a mentor as early as possible. For Outreachy, the application system is already open and will close March 30, 2017 (but you can edit and modify proposals submitted in agreement with your mentor until April 28th).
  • Selection Period: After applying to participate, our mentors will chose the most promising candidates. Successful candidates will be announced on the following dates: April 28 (Outreachy), May 4 (GSoC).
  • Internship Duration: May 30 to August 29 (GSoC) and August 30 (Outreachy).

For a list of projects for participants and more information on how to apply, check our Xen Project 2017 Summer Internship Portal. We have many different projects in many different areas: from Hypervisor work, projects in Mirage OS, to tools and test related tasks. Note that we will be adding extra projects to this page in the coming weeks and that applicants can suggest projects on their own.

You may also want to check out the pages of GSoC mentoring organisations which we collaborate with. Sometimes, you will find Xen related projects there: FreeBSD (currently 2 projects), QEMU, Libvirt.

Learn about the Experience of past Participants

At a past Xen Project Developer Summit, we ran a panel discussion that included Outreachy interns, GSoC students as well as mentors.


You may also want to read Women interns rocking open source at Xen Project.