Tag Archives: embedded

A Recap of the Xen Project Developer and Design Summit: Community Health, Development Trends, Coding Changes and More

We were extremely thrilled to host our Xen Project Developer and Design Summit in Nanjing Jiangning, China this June. The event brought together our community and power users under one roof to collaborate and to learn more about the future of our project. It also gave us the opportunity to connect with a large group of our community who is based in China. We’ve seen a steady stream of Xen Project hypervisor adoption in this region.

If you were unable to attend the event, we have recordings of the presentations, and we also have the slideshares from the presentation available. Please check them out!

During our event, we always start with a weather report on the Xen Project. It covers areas that we are improving upon, where we need more support, and also the potential direction of the project. This blog covers information from the weather report as well as next steps and focus areas for the project.

Community Health
Code commits for the hypervisor have on average grown by 11% YoY since 2014. Commits in the first 5 months of 2018 have grown 11% compared to the same period last year. The top 6 contributors to the project since 2011 have been Citrix, Suse, AMD, Arm, Intel, Oracle. This is also true for the last 12 months in which 90% of contributions came from the top 6 players.

However, we have seen a larger than normal volume of contributions from Arm and AMD, which contributed twice as much as in previous years. In addition, EPAM is establishing itself on the top table with first contributions and a significant number of code reviews.  In addition, AWS started to make first significant code contributions in 2017.

Hardware security issues had an impact on the code review process of the project and thus on the project’s capability to take in some code. In other words, x86 related development that was not directly attributed to hardware security issues were slowed down, because developers normally reviewing contributions had less bandwidth to do so.

This has forced the community to make some changes that are starting to have a positive effect: x86 developers across companies are collaborating more and better, meaning that hardware security issues in 2018 had a smaller impact on the community than those in 2017.

Innovation and Development Trends
Unikraft, a Xen Project sub-project, is on a healthy growth projection. Unikraft aims to simplify the process of building unikernels through a unified and customizable code base. It was created after Xen Project Developer and Design Summit 2017.

The project recently upstreamed a significant amount of functionality, including:

  • Scheduling support, better/more complete support for KVM/Xen/Linux. Supporting Xen/KVM allows Unikraft to cater to a larger set of potential users/companies. Linux user-space provides an excellent development environment: Unikraft users can create their Unikraft unikernels as a Linux executable, use Linux’s wide range of debugging and performance optimization tools, and when done simply re-compile as a KVM or Xen unikernel (work on creating x86/Arm bare metal images is ongoing).
  • A release of newlib (a libc-like library) and lwip (a network stack: This support allows Unikraft to compile with most applications. It is a basic requirement to support a potentially wide range of applications.
  • The project is beginning to pick up traction with contributions coming from companies like NEC, Arm, and Oracle.

For more information check out the following two presentations: Unikraft and Unikraft on Arm.

We have been re-writing the x86 core. We are working on adding complex new CPU hardware features such as support for NVDIMMs and SGX. In addition, we are working on making technologies that have been used by security-conscious vendors in non-server environments ready to be used in server virtualization and cloud computing; support for measured boot is an example.

Another key innovation is a project called Panopticon, which aims to re-write some portions of the hypervisor to make Xen resilient to all types of side-channel attacks by removing unnecessary information about guests from the hypervisor.

You can find presentations related to these topics here (x86 evolution) and here (side-channel attacks and mitigations).

Continued Growth in Embedded and Automotive
We are seeing continued contributions within the embedded and automotive space to Xen Project Core with new features and functionality, including:

  • Co-processor (GPU) sharing framework enabling virtualization of co-processors such as FPGAs, DRMs, etc.
  • 2nd generation Power management and HPM on Arm  – this enables a huge reduction in power consumption, which is significant for some embedded market segments.
  • RTOS based Dom0 and code size reduction – this reduces the cost of safety certification significantly and is important for market segments where safety certification is important (such as automotive, avionics, medical, etc). We already managed to get Xen code size on Arm to below 45K SLOC and we expect that Dom0 will also be below 50K SLOC. This makes it possible to safety certify a Xen based stack to DAL C ASIL-B/C standards at a cost equivalent to less than 10 years.
  • Improved startup latency to boot multiple VMs in parallel from the device tree – this opens up the use of Xen to small IoT and embedded devices and allows booting of a complete Xen system in milliseconds compared to seconds. In addition, it halves the cost of safety certifications for systems where a Dom0 is not necessary

You can see the progress of our re-architecture in our latest release, Xen Project hypervisor 4.11. Also, the following summit presentations were relevant: here (Xen and automotive at Samsung) here (CPUFreq) and here (Real-time support).

These are just a few features and updates that make it easier for Xen to be used in embedded environments and market segments where safety certification is relevant. In addition, this will also significantly improve BoM and security in other market segments. On x86 we are also reducing code size, but this is significantly harder because of backward compatibility guarantees for x86 hardware and older operating systems.

Conclusion
The event was a great success with a lot of community and technical topics, like “How to Get Your Code Into Xen” and “The Art of Virtualizing Cache Maintenance.” Find the playlist for the full conference here. Additionally, our design sessions focused on architecture, embedded and safety, security, performance, and working practices and processes. You can find what was discussed, and next steps with these areas on our wiki.

If you want to stay abreast of where and when the Xen Project Developer and Design Summit will be held next year, follow us on Facebook and Twitter.

Xen Project Matrix

Xen Project Hypervisor: Virtualization and Power Management are Coalescing into an Energy-Aware Hypervisor

Power management in the Xen Project Hypervisor historically targets server applications to improve power consumption and heat management in data centers reducing electricity and cooling costs. In the embedded space, the Xen Project Hypervisor faces very different applications, architectures and power-related requirements, which focus on battery life, heat, and size.

Although the same fundamental principles of power management apply, the power management infrastructure in the Xen Project Hypervisor requires new interfaces, methods, and policies tailored to embedded architectures and applications. This post recaps Xen Project power management, how the requirements change in the embedded space, and how this change may unite the hypervisor and power manager functions. Read the full article on Linux.com here.

Join us at Root Linux Conference Happening in Kyiv, Ukraine This April!

Root Linux Conference is coming to Kyiv, Ukraine on April 14th. The conference is the biggest Linux and embedded conference in Eastern Europe with presenters exploring topics like: Linux in mobile devices, wearables, medical equipment, vehicles, and more. Want to learn about the next generation of embedded solutions? This is the conference for you.

Juergen Gross, Linux Kernel developer at SUSE, and Paul Durrant, Senior Principal Software Engineer at Citrix Systems, are keynoting the conference. Juergen will cover Xen paravirtualized (PV) devices and Paul will cover Intel GVT-g integration into XenServer.

Early bird priced tickets for the event are still available here.

See you there!

Call for Proposals Open for the Xen Project Developer and Design Summit Happening in June!

Registration and the call for proposals are open for the Xen Project Developer and Design Summit 2018, which will be held in Nanjing Jiangning, China from June 20 – 22, 2018. The Xen Project Developer and Design Summit combines the formats of Xen Project Developer Summits with Xen Project Hackathons, and brings together the Xen Project’s community of developers and power users.

Submit a Talk

Do you have an interesting use case around Xen Project technology or best practices around the community? There’s a wide variety of topics we are looking for, including cloud, server virtualization, unikernels, automotive, security, embedded environments, network function virtualization (NFV), and more. You can find all the suggested topics for presentations and panels here (make sure you select the Topics tab).

Several formats are being accepted for speaking proposals, including:

  • Presentations and panels
  • Interactive design and problem solving sessions. These sessions can be submitted as part of the CFP, but we will reserve a number of design sessions to be allocated during the event.
    • Proposers of design sessions are expected to host and moderate design sessions following the format we have used at Xen Project Hackathons. If you have not participated in these in the past, check out past event reports from 2016, 2015 and 2013.

Never talked at a conference before? Don’t worry! We encourage new speakers to submit for our events!

Here are some dates to remember for submissions and in general:

  • CFP Close: April 13, 2018
  • CFP Notifications: April 30 – May 2, 2018
  • Schedule Announced: May 3, 2018
  • Event: June 20 – 22, 2018

Registration

Come join us for this event, and if you register by May 2, you’ll get an early bird discount of $125/ 800 Yuan Travel stipends are available for students or individuals that are not associated with a company. If you have any questions, please send a note to community.manager@xenproject.org.

Curious about last year’s event? Check out a few of our presentations last year here!

Xen on ARM interrupt latency

Xen on ARM is becoming more and more widespread in embedded environments. In these contexts, Xen is employed as a single solution to partition the system into multiple domains, fully isolated from each other, and with different levels of trust.

Every embedded scenario is different, but many require real-time guarantees. It comes down to interrupt latency: the hypervisor has to be able to deliver interrupts to virtual machines within a very small timeframe. The maximum tolerable lag changes on a case by case basis, but it should be in the realm of nanoseconds and microseconds, not milliseconds.

Xen on ARM meets these requirements in a few different ways. Firstly, Xen comes with a flexible scheduler architecture. Xen includes a set of virtual machine schedulers, including RTDS, a soft real-time scheduler, and ARINC653, a hard real-time scheduler. Users can pick the ones that perform best for their use-cases. However, if they really care about latency, the best option is to have no schedulers at all and use a static assignment for virtual cpus to physical cpus instead. There are no automatic ways to do that today, but it is quite easy to achieve with the vcpu-pin command:

Usage: xl vcpu-pin [domain-id] [vcpu-id] [pcpu-id]

For example, in a system with four physical cpus and two domains with two vcpus each, a user can get a static configuration with the following commands:

xl vcpu-pin 0 0 0
xl vcpu-pin 0 1 1
xl vcpu-pin 1 0 2
xl vcpu-pin 1 1 3

As a result, all vcpus are pinned to different physical cpus. In such a static configuration, the latency overhead introduced by Xen is minimal. Xen always configures interrupts to target the cpu that is running the virtual cpu that should receive the interrupt. Thus, the overhead is down to just the time that it takes to execute the code in Xen to handle the physical interrupt and inject the corresponding virtual interrupt to the vcpu.

For my measurements, I used a Xilinx Zynq Ultrascale+ MPSoC, an excellent board with four Cortex A53 cores and a GICv2 interrupt controller. I installed Xen 4.9 unstable (changeset 55a04feaa1f8ab6ef7d723fbb1d39c6b96ad184a) and Linux 4.6.0 as Dom0. I ran tbm as a guest, which is a tiny baremetal application that programs timer events in the future, then, after receiving them, checks the current time again to measure the latency. tbm uses the virtual timer for measurements, however, the virtual timer interrupt is handled differently compared to any other interrupts in Xen. Thus, to make the results more generally applicable, I modified tbm to use the physical timer interrupt instead. I also modified Xen to forward physical timer interrupts to guests.

Keeping in mind that the native interrupt latency is about 300ns on this board, these are the results on Xen in nanoseconds:

AVG  MIN  MAX  WARM_MAX
4850 4810 7030 4980

AVG is the average latency, MIN is the minimum, MAX is the maximum and WARM_MAX is the maximum latency observed after discarding the first few interrupts to warm the caches. For real-time considerations, the number to keep in mind is WARM_MAX, which is 5000ns (when using static vcpu assignments).

This excellent result is small enough for most use cases, including piloting a flying drone. However, it can be further improved by using the new vwfi Xen command line option. Specifically, when vcpus are statically assigned to physical cpus using vcpu-pin, it makes sense to pass vwfi=native to Xen: it tells the hypervisor not to trap wfi and wfe commands, which are ARM instructions for sleeping. If no other vcpus can ever be scheduled on a given physical cpu, then we might as well let the guest put the cpu to sleep. Passing vwfi=native, the results are:

AVG  MIN  MAX  WARM_MAX
1850 1680 2650 1950

With this configuration, the latency is only 2 microseconds, which is extremely close to the hardware limits, and should be small enough for the vast majority of use cases. vwfi was introduced recently, but it has been backported to all the Xen stable trees.

In addition to vcpu pinning and vwfi, the third key parameter to reduce interrupt latency is unexpectedly simple: the DEBUG kconfig option in Xen. DEBUG is enabled by default in all cases except for releases. It adds many useful debug messages and checks, at the cost of increased latency. Make sure to disable it in production and when doing measurements.