Xen on ARM interrupt latency

Xen on ARM is becoming more and more widespread in embedded environments. In these contexts, Xen is employed as a single solution to partition the system into multiple domains, fully isolated from each other, and with different levels of trust.

Every embedded scenario is different, but many require real-time guarantees. It comes down to interrupt latency: the hypervisor has to be able to deliver interrupts to virtual machines within a very small timeframe. The maximum tolerable lag changes on a case by case basis, but it should be in the realm of nanoseconds and microseconds, not milliseconds.

Xen on ARM meets these requirements in a few different ways. Firstly, Xen comes with a flexible scheduler architecture. Xen includes a set of virtual machine schedulers, including RTDS, a soft real-time scheduler, and ARINC653, a hard real-time scheduler. Users can pick the ones that perform best for their use-cases. However, if they really care about latency, the best option is to have no schedulers at all and use a static assignment for virtual cpus to physical cpus instead. There are no automatic ways to do that today, but it is quite easy to achieve with the vcpu-pin command:

Usage: xl vcpu-pin [domain-id] [vcpu-id] [pcpu-id]

For example, in a system with four physical cpus and two domains with two vcpus each, a user can get a static configuration with the following commands:

xl vcpu-pin 0 0 0
xl vcpu-pin 0 1 1
xl vcpu-pin 1 0 2
xl vcpu-pin 1 1 3

As a result, all vcpus are pinned to different physical cpus. In such a static configuration, the latency overhead introduced by Xen is minimal. Xen always configures interrupts to target the cpu that is running the virtual cpu that should receive the interrupt. Thus, the overhead is down to just the time that it takes to execute the code in Xen to handle the physical interrupt and inject the corresponding virtual interrupt to the vcpu.

For my measurements, I used a Xilinx Zynq Ultrascale+ MPSoC, an excellent board with four Cortex A53 cores and a GICv2 interrupt controller. I installed Xen 4.9 unstable (changeset 55a04feaa1f8ab6ef7d723fbb1d39c6b96ad184a) and Linux 4.6.0 as Dom0. I ran tbm as a guest, which is a tiny baremetal application that programs timer events in the future, then, after receiving them, checks the current time again to measure the latency. tbm uses the virtual timer for measurements, however, the virtual timer interrupt is handled differently compared to any other interrupts in Xen. Thus, to make the results more generally applicable, I modified tbm to use the physical timer interrupt instead. I also modified Xen to forward physical timer interrupts to guests.

Keeping in mind that the native interrupt latency is about 300ns on this board, these are the results on Xen in nanoseconds:

4850 4810 7030 4980

AVG is the average latency, MIN is the minimum, MAX is the maximum and WARM_MAX is the maximum latency observed after discarding the first few interrupts to warm the caches. For real-time considerations, the number to keep in mind is WARM_MAX, which is 5000ns (when using static vcpu assignments).

This excellent result is small enough for most use cases, including piloting a flying drone. However, it can be further improved by using the new vwfi Xen command line option. Specifically, when vcpus are statically assigned to physical cpus using vcpu-pin, it makes sense to pass vwfi=native to Xen: it tells the hypervisor not to trap wfi and wfe commands, which are ARM instructions for sleeping. If no other vcpus can ever be scheduled on a given physical cpu, then we might as well let the guest put the cpu to sleep. Passing vwfi=native, the results are:

1850 1680 2650 1950

With this configuration, the latency is only 2 microseconds, which is extremely close to the hardware limits, and should be small enough for the vast majority of use cases. vwfi was introduced recently, but it has been backported to all the Xen stable trees.

In addition to vcpu pinning and vwfi, the third key parameter to reduce interrupt latency is unexpectedly simple: the DEBUG kconfig option in Xen. DEBUG is enabled by default in all cases except for releases. It adds many useful debug messages and checks, at the cost of increased latency. Make sure to disable it in production and when doing measurements.

Now Accepting Submissions for Xen Project Developer and Design Summit 2017

We’re excited to announce that registration and the call for proposals is open for Xen Project Developer and Design Summit 2017, which will be held in Budapest, Hungary from July 11-13, 2017. The Xen Project Developer and Design Summit combines the formats of Xen Project Developer Summits with Xen Project Hackathons, and brings together the Xen Project’s community of developers and power users.

Submit a Talk

Do you have an interesting use case around Xen Project technology or best practices around the community? There’s a wide variety of topics we are looking for, including security, embedded environments, network function virtualization (NFV), and more. You can find all the suggested topics for presentations and panels here (make sure you select the Topics tab).

Several formats are being accepted for speaking proposals, including:

  • Presentations and Panels
  • Interactive design and problem solving sessions. These sessions can be submitted as part of the CFP, but we will reserve a number of design sessions to be allocated during the event. Proposers of design sessions are expected to host and moderate design sessions following the format we have used at Xen Project Hackathons. If you have not participated in these in the past, check out past event reports from 2016, 2015 and 2013.

Never talked at a conference before? Don’t worry! We encourage new speakers to submit for our events and have plenty of resources to help you prepare for your presentation.

Here are some dates to remember for submissions and in general:

  • CFP Close: April 14, 2017
  • CFP Notifications: May 5, 2017
  • Schedule Announced: May 16, 2017
  • Event: July 11-13, 2017


Come join us for this event, and if you register by May 19, you’ll get an early bird discount :) Travel stipends are available for students or individuals that are not associated with a company. If you have any questions, please send a note to community.manager@xenproject.org.

Xen Project Maintenance Releases Available (Versions 4.6.5 and 4.7.2)

I am pleased to announce the release of Xen 4.6.5 and 4.7.2. Xen Project Maintenance releases are released in line with our Maintenance Release Policy. We recommend that all users of the 4.6 and 4.7 stable series update to the latest point release.

Xen 4.6.5

Xen 4.6.5 is available immediately from its git repository http://xenbits.xen.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.6
(tag RELEASE-4.6.5) or from the Xen Project download page http://www.xenproject.org/downloads/xen-archives/supported-xen-46-series/xen-465.html

Xen 4.7.2

Xen 4.7.2 is available immediately from its git repository http://xenbits.xen.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.7
(tag RELEASE-4.7.2) or from the Xen Project download page http://www.xenproject.org/downloads/xen-archives/supported-xen-47-series/xen-472.html

These releases contain many bug fixes and improvements. For a complete list of changes, please check the lists of changes on the download pages.

Xen Project Participates in Google Summer of Code and Outreachy

This is a quick announcement that the Xen Project is again participating in Google Summer of Code (GSoC), a program that awards three-month paid stipends to University students to work on open source projects, with the goal to get open source experience. The Xen Project will also again participate in Outreachy, which is an internship program that organises three-month paid internships with free and open-source software projects for people who are typically underrepresented in the technology industry. Outreachy has been helping women (cis and trans), trans men, genderqueer people and people from other discriminated backgrounds get involved in free and open source software for several years. The Xen Project is proud that it has participated continually in Outreachy (and its predecessor OPW) for 4 years.

I want to participate, how do I get started?

If you are not at all familiar with programs such as GSoC and Outreachy, have a quick look at our introduction. In a nutshell, both programs go through several stages:

  • Check eligibility requirements.
  • Now until application period: Preparation by working on small tasks (also called micro-tasks) within our community to identify a suitable project and to familiarise yourself with the technology.
  • Application Period (aka paperwork): For GSoC, the application system is open from March 20 to Apr 3, 2017; however you should work on micro-tasks before and prepare your application together with a mentor as early as possible. For Outreachy, the application system is already open and will close March 30, 2017 (but you can edit and modify proposals submitted in agreement with your mentor until April 28th).
  • Selection Period: After applying to participate, our mentors will chose the most promising candidates. Successful candidates will be announced on the following dates: April 28 (Outreachy), May 4 (GSoC).
  • Internship Duration: May 30 to August 29 (GSoC) and August 30 (Outreachy).

For a list of projects for participants and more information on how to apply, check our Xen Project 2017 Summer Internship Portal. We have many different projects in many different areas: from Hypervisor work, projects in Mirage OS, to tools and test related tasks. Note that we will be adding extra projects to this page in the coming weeks and that applicants can suggest projects on their own.

You may also want to check out the pages of GSoC mentoring organisations which we collaborate with. Sometimes, you will find Xen related projects there: FreeBSD (currently 2 projects), QEMU, Libvirt.

Learn about the Experience of past Participants

At a past Xen Project Developer Summit, we ran a panel discussion that included Outreachy interns, GSoC students as well as mentors.

You may also want to read Women interns rocking open source at Xen Project.

How To Shrink Attack Surfaces with a Hypervisor

A software environment’s attack surface is defined as the sum of points in which an unauthorized user or malicious adversary can enter or extract data. The smaller the attack surface, the better. Linux.com recently sat down with Doug Goldstein (https://github.com/cardoe or @doug_goldstein) to discuss how companies can use hypervisors to reduce attack surfaces and why the Xen Project hypervisor is a perfect choice for security-first environments. Doug is a principal software engineer at Star Lab, a company focused on providing software protection and integrity solutions for embedded systems.

You can read the full interview here.

Request for Comment: Scope of Vulnerabilities for which XSAs are issued

Issuing advisories has a cost: It costs the security team significant amounts of time to craft and send the advisories; it costs many of our downstreams time to apply, build, and test patches; and it costs many of our users time to decide whether to do an update, and if so, to test and deploy it.

Given this, the Xen Project Security Team wants to clarify when they should issue an advisory or not: the Xen Security Response Process only mentions “‘vulnerabilities”, without specifying what constitutes a vulnerability.

We would like guidelines from the community about what sorts of issues should be considered security issues (and thus will have advisories issued). I have posted the second version a draft of a section I am proposing to be added to the Xen Security Policy to xen-devel; a copy is included below for your convenience. There are only minor modifications from the first draft, so barring major feedback from the wider community it will likely achieve consensus. If you want input, now is the time to speak up.

Most of it is just encoding long-established practice. But there are two key changes and / or clarifications that deserve attention and discussion:

    Criteria 2c: Leaking of mundane information from Xen or dom0 will not be considered a security issue unless it may contain sensitive guest or user data

Criteria 4: If no operating systems are vulnerable to a bug, no advisory will be issued.

If you want to weigh in on the question, please join the discussion on xen-devel before 28 February. The title of the thread is “RFC v2: Scope of Vulnerabilities for which XSAs are issued”.

Continue reading