Tag Archives: CES

CES 2015: Smart Cars are the New Smart Phone

This is a reprint of the following Linux.com article by Alex Agizim, VP, CTO Embedded Systems at GlobalLogic

“Smart car” technology had a huge presence at CES 2015, from BMW’s 360-degree collision avoidance and parking assist features to Audi’s Human Machine Interface (HMI) that connects to an iPhone or Android device. And with both Apple and Google jumping into the market with their CarPlay and Android Auto IVI systems, the automotive industry is on the brink of some significant changes.

For example, thanks to new developments in open source virtualization, OEMs and car manufacturers are closer than ever to achieving a secure, flexible, robust, and customizable integrated cockpit — one that keeps drivers safe while meeting consumers’ connected car expectations. Already well-known for providing security, stability, and isolation in the datacenter, automotive virtualization is gaining wider attention due to additional hardening and new support for ARM.

While this is certainly exciting, virtualization remains a roadblock to some in the smart car industry. I personally had the opportunity to demonstrate GlobalLogic’s Nautilus platform for automotive virtualization at GENIVI’s CES demo and networking event. Leveraging a TI J6 SoC, I demo’d a dual-screen virtual cockpit with one screen emulating a Linux-powered driver information display, and the other screen emulating an Android-powered IVI system. The entire configuration ran on Xen Project Hypervisor 4.5 with three domains: Dom0 (thin control), DomU (Linux), and DomU (Android).

During the demo, I showcased how Nautilus achieves an overall system boot time of 8 seconds, an early RVC of 1.5 seconds, and secure and reliable peripheral sharing (including GPUs). Most importantly, I demonstrated how even if the Android virtual machine crashes, it has absolutely no influence on the mission-critical Linux virtual machine. With Nautilus automotive software, developers can host a number of VMs that are completely sandboxed from each other, thereby ensuring that all vehicle services will continue to operate even if one specific component fails.

The demo was well-received by GENIVI’s attendees, and I got the impression that many Tier 1 OEMs were thinking about using virtualization in their next-gen platforms. This is a huge milestone because, up until very recently, virtualization had a bad rep in the automotive industry. Previous attempts at virtualization using ARM A9 architecture ultimately failed because there was no hardware support for it. Many were also highly reluctant to use open source technology because it lacked proper compliance to strict auto industry regulations. But with platforms like Nautilus, developers can leverage cutting-edge open source technology that is ISO 26262 certification ready to create secure and reliable automotive virtualization experiences.

In fact, GlobalLogic’s goal is to make Nautilus part of the reference Automotive Grade Linux (AGL) software, an open source project that is developing a common, Linux-based software stack for the connected car. We are also a founding leader for Xen Project’s Embedded and Automotive initiative. GlobalLogic is working to add the Xen-based technology to the AGL spec and is further developing the platform’s real-time scheduling and peripheral sharing features to improve the use of a single physical CPU for multiple guest OSes and peripheral devices. We’ll soon be extending QNX and Tizen IVI 3.0 support to improve the functionality of other features. Finally, we are also expanding Nautilus to support even more SoCs in the next six months, such as Renesas R-Car H2/M2, which offers hardware support for virtualization.

Based on my work with the Nautilus platform and my observations of the general automotive industry, I wouldn’t be surprised to see the first PoCs for automotive virtualization coming out of China and Japan later this year. The momentum behind smart car technology development is very strong right now, and I’m excited to see what happens when automotive OEMs finally start taking advantage of virtualization’s many possibilities.

Will you give Xen a ride … or will Xen give you a ride?

And by “a ride”, we actually mean a ride. Like this:



Like, will Xen run in your car?  Well, it appears it will!

It all started with ARM Support

In fact, Xen Project developers started woking on supporting the ARM architecture (with hardware virtualization capabilities) a couple of years ago. The goal was simple: as soon as ARM server are available, it must be possible to run Xen Project software on them. That goal has been achieved, but that is another story!

It is well known that processors employing the ARM architecture are powering already the vast majority of the so called Embedded Systems, ranging from phones, tablets and smart TVs up to cars or even airplanes. But does that mean that at some point we will start to see virtualization capable chips in cars? And if yes, when? The answers to these questions are “Yes” and “really really really soon”! In fact, the Xen Project Hypervisor is uniquely placed to support this new range of use-cases. Its isolation and security features, flexible virtualization mode and architecture, not to mention driver disaggregation and the fact that it now supports ARM (and does it with only ~90K lines of code), make it a perfect fit for the embedded world.

Some Recent ‘History’

Mobile and embedded virtualization on ARM has a long history within the Xen Project, with research projects such as Samsung’s ARM PV port and the Embedded Xen effort. However these projects were mainly research focused. With ARM support becoming a part of the Xen Project Hypervisor last year and various market factors coming together, Xen Project based products are now on the horizon. Last autumn was pivotal in generating momentum for this concept. A number of companies showed real demos and prototypes at our 2013 Developer Summit, such as

  • The Xen Project Hypervisor running on a Nexus 10 (slides and video)
  • The Xen Project Hypervisor powering an in-vehicle infotainment (IVI) system, and other systems on the TI Jacinto 6 automotive platform designed for cars (slides and video).

Since then, momentum has built within the community – as can be seen on xen-devel mailing list discussions – to port embedded OSes to the Xen Hypervisor  (some examples: FreeRTOS, Erika and QNX). Contributions and patches for making The Xen Project Hypervisor work better in such environments started to arrive too, from individuals, research institutions and small and big companies. Among the companies, GlobalLogic Inc., a full-lifecycle product development services company, has made the largest contribution so far, but we must also mention DornerWorks, GaloisUniversity of Washington and Evidence (in collaboration with the University of Modena).

A summary of the past and ongoing activities of this kind is below:

What about now?

On Monday (we told you: “really really really soon” :-D), The Xen Collaborative Project and The Linux Foundation announced a new Embedded and Automotive initiative. Artem Mygaiev, AVP Development at GlobalLogic, will serve as the Embedded and Automotive Project Lead.

The Embedded and Automotive team within The Xen Project intends to build a platform around the Xen Hypervisor that enables using it for all the non-data center use cases (automotive, internet TV, mobile, etc.) by providing a community focal-point within the Xen Project community as well as within the wider open source community.

The team plans to:

  • develop and upstream necessary changes to The Xen Project Hypervisor and Linux
  • implement new drivers (such as GPU, HID, …), protocols, capabilities and functionality that are needed for a complete automotive/embedded/mobile virtualization stack
  • upstream all necessary changes to support such functionality in operating systems that are needed for these use-cases (e.g. Android, Linux, etc.)

For the occasion, Alex Agizim, CTO of Embedded Systems at GlobalLogic, which also is a member of The Linux Foundation Automotive Grade Linux Steering Committee, said:

With ARM support, Xen Project technology is a perfect fit for embedded systems and automotive use. For example, our Nautilus platform, based on The Xen Project virtualization, enables ourin-vehicle infotainment (IVI) and auto manufacturing partners to quickly and cost-effectively develop hybrid Android/Linux-based systems. Using Nautilus, developers are able to run multiple sandboxed OSes on a single System-on-Chip (SOC). This provides superior functionality and security for both infotainment and operational functions within a car.

The latest demo of GlobalLogic‘s Nautilus Platform has been shown at the latest edition of the Automotive Linux Summit, in Tokyo. Check out the video and slides. We also heard about further use cases for Xen Project Software at this week’s Developer Summit. The rate of innovation in our community in this area is staggering: fasten your seat belts! We will tell you about these more in an upcoming event report. All this activity is also creating many benefits for the cloud and traditional server use use-cases. Certification will lead to quality improvements across shared components. Realtime scheduling can be used for graphics and gaming use-cases in the cloud and for Network Function Virtualization. And so on, and so on, …

Learn More

GlobalLogic, in partnership with The Linux Foundation, will present a free webinar at 9 a.m. PDT, Wednesday, August 27, 2014, titled “Virtualization in the Automotive Industry.” Register today to learn how Xen Project technology adds reliability and security when adopting virtualization for automotive software development.

Vendors and individual developers interested in collaborating on embedded, automotive and mobile use cases are encouraged to join the new Xen Project subproject at http://xenproject.org/developers/teams/embedded-and-automotive.html.

Summary of the gains of Xen oxenstored over cxenstored

This week, we are reblogging this excellent piece from Luis from SUSE. The article came about because of a discussion Luis had at the Linux Foundation Collaboration Summit in Napa, and he decided to write down some basic generals of the xenstore, a review of its first implementation and a summary of oxenstored. The original post is available here, on Luis’ blog.

OXenstored is the default in Xen, only if the ocaml compiler was installed at build time. This means that in many Linux distributions that ship Xen, cxenstored is most likely your default. You can check whether oxenstored is installed by checking whether the oxenstored process runs in your Dom 0.

It all started with a paper: OXenstored – An Efficient Hierarchical and Transactional Database using Functional Programming with Reference Cell Comparisons

First a general description of the xenstore and its first implementation. The xenstore is where Xen stores the information over its systems. It covers dom0 and guests and it uses a filesystem type of layout kind of how we keep a layout of a system on the Linux kernel in sysfs. The original xenstored, which the paper refers to a Cxenstored was written in C. Since all information needs to be stored in a filesystem layout any library or tool that supports designing a tree to have key value store of information should suffice to upkeep the xenstore. The Xen folks decided to use the Trivial Database, tdb, which as it turns out was designed and implemented by the Samba folks for its own database. Xen then has a daemon sitting in the background which listens to reads / write requests onto this database, that’s what you see running in the background if you ‘ps -ef | grep xen’ on dom0. dom0 is the first host, the rest are guests. dom0 uses Unix domain sockets to talk to the xenstore while guests talk to it using the kernel through the xenbus. The code for opening up a connection onto the c version of the xenstore is in tools/xenstore/xs.c and the the call is xs_open(). The first attempt by code will be to open the Unix domain socket with get_handle(xs_daemon_socket()) and if that fails it will try get_handle(xs_domain_dev()), the later will vary depending on your Operating System and you can override first by setting the environment variable XENSTORED_PATH. On Linux this is at /proc/xen/xenbus. All the xenstore is doing is brokering access to the database. The xenstore represents all data known to Xen, we build it upon bootup and can throw it out the window when shutting down, which is why we should just use a tmpfs for it (Debian does, OpenSUSE should be changed to it). The actual database for the C implementation is by default stored under the directory /var/lib/xenstored, the file that has the database there is called tdb. On OpenSUSE that’s /var/lib/xenstored/tdb, on Debian (as of xen-utils-4.3) that’s /run/xenstored/tdb. The C version of the xenstore therefore puts out a database file that can actually be used with tdb-tools (actual package name for Debian and SUSE). xentored does not use libtdb which is GPLv3+, Xen in-takes the tdb implementation which is licensed under the LGPL and carries a copy under tools/xenstore/tdb.c. Although you shouldn’t be using tdb-tools to poke at the database you can still read from it using these tools, you can read the entire database as follows:

 tdbtool /run/xenstored/tdb dump

The biggest issue with the C version implementation and relying on tdb is that you can live lock it if you have a guest or any entity doing short quick accesses onto the xenstore. We need Xen to scale though and the research and development behind oxenstored was an effort to help with that. What follows next is my brain dump of the paper. I don’t get into the details of the implementation because as can be expected I don’t want to read OCaml code. Keep in mind that if I look for a replacement I’m looking also for something that Samba folks might want to consider.

OXenstored has the following observed gains:

  • 1/5th the size in terms of line of code in comparison to the C xenstored
  • better performance increasing support for the number of guests, it supports 3 times number of guests for an upper limit of 160 guests

The performance gains come from two things:

  • how it deals with transactions through an immutable prefix tree. Each transaction is associated with a triplet (T1, T2, p) where T1 is the root of the database just before a transaction, T2 is the local copy of the database with all updates made by the transaction made up to that point, p is the path to the furthest node from the root T2 whose subtree contains all the updates made by the transaction up that point.
  • how it deals with sharing immutable subtrees and uses ‘reference cell equality’, a limited form of pointer equality, which compares the location of values instead of the values themselves. Two values are shared if they share the same location. Functional programming languages enforce that multiple copies of immutable structures share the same location in memory. oxenstored takes avantage of this functional programming feature to design trie library which enforces sharing of  subtrees as much as possible. This lets them simpilfy how to determine and merge / coalesce concurrent transactions.

The complexity of the algorithms used by oxenstored is confined only to the length of the path, which is rarely over 10. This gives predictable performance regardless of the number of guests present.

Ballooning, rebooting, and the feature you’ve never heard of

Today I’d like to talk about a functionality of Xen you may not have heard of, but might have actually used without even knowing it. If you use memory ballooning to resize your guests, you’ve likely used “populate-on-demand” at some point. 

As you may know, ballooning is a technique used to dynamically adjust the physical memory in use by a guest. It involves having a driver in the guest OS, called a balloon driver, allocate pages from the guest OS and then hand those pages back to Xen. From the guest OS perspective, it still has all the memory that it started with; it just has a device driver that’s a real memory hog. But from Xen’s perspective, the memory which the device driver asked for is no longer real memory — it’s just empty space (hence “balloon”). When the administrator wants to give memory back to the VM, the balloon driver will ask Xen to fill the empty space with memory again (shrinking or “deflating” the balloon), and then “free” the resulting pages back to the guest OS (making the memory available for use again).

While this can be used to shrink guest memory and then expand it again, this technique has an important limitation: It can never grow the memory above the starting size of the VM. This is because the only way to grow guest memory is to “deflate” the balloon. Once it gets back to the starting size of the VM, the balloon is entirely deflated and no additional memory can be added by the balloon driver.

To see why this is important, consider the following scenario.

Host A and B both have 4GiB of RAM, and 2 VMs with 2GiB of RAM each. Suppose you want to reboot host B to do some hardware maintenance. You could do the following:

  • Balloon all 4 VMs down to 1GiB
  • Migrate the 2 VMs from host B onto host A
  • Shut down host B to do your maintenance
  • Bring up host B
  • Migrate the 2 VMs originally on host B back
  • Balloon all 4 VMs back up to 2GiB

All well and good. But suppose that while you had one of those VMs ballooned down to 1GiB, you needed to reboot one. Now you have a problem: Most operating systems will only check how much memory is available at boot time. You only have 1GiB of free memory. If you boot with 1GiB of memory, you will be able to balloon *smaller* than 1GiB, but you will not be able to balloon back up to 2GiB when the maintenance of host B is done.

This is where populate-on-demand comes in. It allows a VM to boot with a maximum memory larger than its current target memory. It enables a guest that thinks it has 2GiB of RAM to boot while only actually using 1GiB of RAM. It can do this because it only needs to allow the guest to run until the balloon driver can start. Once the balloon driver starts, it will “inflate” the balloon to the proper size. At that point, there is nothing special to do; the VM looks like it did when we shut it down (guest thinks it has 2GiB of RAM, but 1GiB is allocated to the balloon driver and not accessed). When host B comes back up and more memory is free, the balloon driver can deflate the balloon, bringing the total memory back up to 2GiB.

Populate-on-demand comes into play in Xen whenever you start an HVM guest with maxmem and memory set to different values. In that case, the guest will be told it has maxmem RAM, but will only have memory allocated to it; the populate-on-demand code will allow the guest to run in this mode until the balloon driver comes up and hands “frees” maxmem-memory back to Xen.

Virtualizing memory: A primer

In order to desrcibe how populate-on-demand works, I’ll need to explain a bit more about how Xen virtualizes memory. On real hardware, the actual hardware memory is referred to as physical memory; and it is typically divided into 4k-chunks called physical frames. These frames are addressed by their physical frame number, or pfn. In the x86 world, pfns typically start at 0, and are mostly contiguous (with the occasional “hole” for IO devices). Historically, on x86 platforms, a description of which pfns are available for use by memory is in something called the E820 map, provided by the BIOS to operating systems at boot.

When we virtualize, we need to provide the guest with virtual “physical address space,” described in the virtual E820 map provided to the guest. These are called guest physical frame numbers, or gpfns. But of course there is still real hardware backing this memory; in the virtualization world, it is common to refer to these as machine frames, or mfns. Every useable gpfn must have a mfn behind it.

But the gpfns have to start at 0 and be contiguous, while the mfns which back them may come from anywhere in Xen’s memory. So every VM has a physical to machine translation table, or p2m table, which maps the gpfn space onto the mfn space. Each gpfn will have an entry in the table, and every useable bit of RAM has an mfn behind it. Normally this is done by the domain builder in domain 0, which will ask Xen to fill the p2m table appropriately (including any holes for IO devices if necessary).

Ballooning then works like this. To inflate the balloon, the balloon driver will ask the guest OS for a free page. After allocating the page, it puts it on its list of pages and finds the gpfn for that page. It then tells Xen it can take the memory behind the gpfn back. Xen will replace the mfn in that gpfn space with “invalid entry,” and put the mfn on its own free list (available to be given to another VM). If the guest were to attempt to read or write this memory now, it would crash; but it won’t, because the guest OS thinks the page is in use by the balloon driver. The balloon driver won’t touch it, and the OS won’t use it for anything else.

To deflate the balloon, the balloon driver will choose one of the pages on its list that it has allocated, and then asks Xen to put some memory behind the gpfn. If Xen determines that the guest is allowed to increase its memory, and there is free memory available, then it will allocate an mfn and put it in the p2m table behing that gpfn. Now the gpfn is useable again; the balloon driver then frees the page back to the guest OS, which will put it on its own free list to use for whatever needs memory.

Populate on Demand: The Basics

The idea behind populate-on-demand was that the guest didn’t actually need all of its memory to boot up until the balloon driver was active — it only needed a small portion of it. But there was no way for the domain builder to know ahead of time which gpfns the guest OS will actually need to use in order to do that; nor which memory will be given to the balloon driver by the guest OS once it starts up.

So when building a domain in populate-on-demand mode the domain builder tells Xen to allocate the mfns into a special pool, which I will call here the PoD pool, according to how much memory is specified in the memory parameter. (In the Xen code it’s actually called the PoD cache, but it’s not a good name, because in computer science “cache” has a very specific meaning that doesn’t match what the PoD pool does. This will probably be renamed at some point for clarity.)

It then creates the guest’s p2m table as it did before, but instead of filling it with mfns, it fills it with a special PoD entry. The PoD entry is an invalid entry; so as the guest boots, whenever it touches a gpfn backed by a PoD entry, it will trap up into Xen. When Xen sees that the PoD entry, it will take an mfn from the PoD pool and put it in the p2m for that gpfn. It will then return to the guest, at which point the memory access will succeed and the guest can continue.

Thus, rather than populating the p2m table when building the domain, the p2m table is populated on demand; hence the name.

The key reason for having the the PoD pool is that the memory is already allocated to the domain. If you do a domain list it shows up as owned by the domain; and it cannot be allocated to a different domain. If this were instead allocate on demand, where you actually allocated the memory from Xen when you hit an invalid entry, there would be a risk that the memory you needed to boot until the balloon driver could run would already have been allocated to a different domain.

However, the guest can’t run like this for long. There are far more PoD entries in the p2m table than there are mfns in the PoD pool — that was the point. But the guest OS doesn’t know that; as far as it’s concerned, it has maxmem to work with. If the balloon driver doesn’t start, nothing will keep it from trying to use all of its memory. If it uses up all the memory in the PoD pool, the next time Xen hits a PoD entry, there won’t be any mfns in the PoD pool to populate the entry with. At that point, Xen would have no choice but to kill the guest.

Getting back to normal: the balloon driver

The balloon driver, like the guest operating system, knows nothing about populate-on-demand. It just knows that it has maxmem gpfn space, and it needs to hand maxmem-memory back to Xen. So it begins allocating pages from the guest operating system, and freeing the gpfns back to Xen.

What Xen does next depends on a few things. Xen keeps track of both the number of PoD entries in the p2m table, and the number of mfns in the PoD pool.

  • If the gpfn is a PoD entry, Xen will simply replace the PoD entry with a normal invalid entry and return. This reduces the number of outstanding PoD entries in the pool.
  • If the gpfn has a real mfn behind it, and the number of PoD entries left in the p2m table is more than the number of mfns in the PoD pool, Xen will replace the entry with an invalid entry, and put the mfn back into the PoD pool. This increases the size of the pool.
  • If the gpfn has a real mfn behind it, but the number of PoD entries left in the p2m table is equal to the number of mfns in the pool, it will put the mfn back on the free list, ready to be used by another domain.

Eventually, the number of outstanding PoD entries is equal to the number of entries in the PoD pool, and the system is now in a stable state. There is no more risk that the guest will touch a PoD entry and not find memory in the pool; and for an active OS, eventually all pages will be touched, and the VM will be the same as one booted not in PoD mode.

It’s never that simple: Page scrubbing

At a high level, that’s the idea behind populate-on-demand. Unfortunately, the real world is often a bit more messy than we would like.

On real hardware, if you do a soft reboot (or if you do some special trick, like spraying the RAM with liquid nitrogen), the memory when the operating system starts may still contain information from a previous boot. The freshly booting operating system has no idea what may be in there: it may be security sensitive information like someone’s taxes or private data keys.

To avoid any risk that information from the previous boot might leak into untrusted programs which might run this time, most operating systems will scrub the memory at boot — that is, fill all the memory with zeros. This also means that drivers can assume that freshly allocated memory will already be zeroed, and not bother doing it themselves. Doing this all at once, at the beginning, allows the operating system to use more efficient algorithms, and also localizes the processor cache pollution.

For an operating system running under Xen this is unnecessary, because Xen will scrub any memory before giving it to the guest (for pretty much the same potential security issue). However, many operating systems which run on Xen — in particular, proprietary operating systems like Windows — don’t know this, and will do their own scrub of memory anyway. Typically this happens very early in boot, long before it is possible to load the balloon driver. This pretty much guarantees that every gpfn will be written to before the balloon driver loads. How does populate on demand deal with that?

The key is that the state of a gpfn after it has been scrubbed by the operating system is the same as the default initial state of a gpfn just populated by the PoD code. This means that after a gpfn has been scrubbed by the operating system, Xen can reclaim the page: it can replace the mfn in the p2m table with a PoD entry, and put the mfn in the PoD pool. The next time the VM touches the page, it will be replaced with a different zero page from the PoD pool; but to the VM it will look the same.

So the populate-on-demand system has a number of zero-page reclaim techniques. The primary one is that when populating a new PoD entry, we look at recently populated entries and see if they are zero, and if they are, we reclaim them. The effect of this is to have each scrubbing thread only have one outstanging PoD page at a time.

If that fails, there is another technique we call the “emergency sweep.” When Xen hits a PoD entry, but the PoD pool is empty, before crashing the guest, it will search through all of guest memory, looking for zeroed pages to reclaim. Because this method is very slow, it is only used as a last resort.


So that’s populate-on-demand in a nutshell. There are more complexities under the hood (like trying to keep superpages together), but I’ll leave those for another day.

Xen Project @ FOSDEM’14: an Event Report

As usual, the first weekend of February (1st & 2nd Feb this year) is FOSDEM weekend. Taking place at “ULB Solbosch Campus, Brussels, Belgium, Europe, Earth”, FOSDEM is the Open Source event of the year. At least for Europe: the website claims that FOSDEM hosts 5,000+ geeks and hackers and 512 lectures!

But it doesn’t stop here: the main wireless network provided (essid FOSDEM) was IPv6 only… as announced in the opening keynote(video at online already!). And in fact it took a while for me to figure out why my Android phone could connect but not get any IP ?!?!

Like 20140202_100927_good last year, I went to FOSDEM to see all the cool stuff and to help man the The Xen Project booth. Although this time I also had a my own talk to deliver. This was only my second FOSDEM, so this may still be my “nubiennes” talking, but it is really hard to describe how big and amazing the event is. Even more so, if you consider it is entirely run and organized by volunteers. And members of the Xen Project helped organize parts of FOSDEM beyond our booth : Ian Jackson and Tim Mackey were Video Volunteers and Lars Kurth helped organize a DevRoom.

It is amazing to have the chance to choose from such a huge amount of super high quality talks and presentations. It is great to see what the most thriving Open Source projects have to show off at their booths, in the exposition area, and to collect some gadgets. Being one of those, The Xen Project was giving away T-Shirts and some other gadgets, for free (some project use FOSDEM to raise funds). And that, like last year, has been quite a success!

T-Shirts 20140201_114724_goodapart, this year we decided we really wanted to do our best to show everyone what Xen is really capable of. That is why we decided to invite community members to host demos. It was an experiment: and it has been a great success! Basically, we invited Xen related projects and or companies that use Xen for their solutions and products, to submit a request for an exclusive slot at the booth. They would then show what they do to FOSDEM attendees. I’m calling it a success because we were fully booked and able to show demos of the following projects: Xen on an Android tablet (a SAMSUNG NEXUS 10), Xen Orchestra, Mirage OS, OSv, Qubes OS and ClickOS. I think we really should do this again next year. I ran the QubesOS demo myself, and it was very pleasant to notice people appreciating what high level of isolation and security this very special Linux distribution provides. It leverages some of the most advanced Xen features, such as stub and driver domains (a couple of people were amazed when I showed them how untrusted PDF conversion works in Qubes! :-D). The presence of two Cloud Operating Systems, MirageOS and OSv (both running on top of Xen, of course) also raised quite some interest. Many attendees were impressed about the responsiveness of SAMSUNG’s solution for virtualizing the GPU in their dual Android tablet demo and the performances and the super quick boot time of ClickOS. Others liked the clean and effective interface of Xen Orchestra. One person even commented that Xen Orchestra looks more impressive than what VMWare provides.

It was great to have the chance to talk with many people that know and use Xen happily and fruitfully, and that were willing to acknowledge all our efforts: technical and not. And of course, to grab a free T-Shirt! Having done pretty much the same last year allows me to run a comparison. There is little doubt that there were more people interested in Xen, and much more awareness about the project doing well and actually expanding (e.g. into embedded and automotive). It was also good, as a developer, to get the chance to talk to users (of any kind) and other members of the Xen community. Like the volunteers showing the demos. For example. I had a great discussion with developers from Samsung about helping them upstream some (at least the kernel and Xen parts) of the incredible work they have done with GPU virtualization on the NEXUS 10.


Even when I wasn’t at the booth, there have been  many chances to hear about Xen, in the various talks given in the Virtualization, BSD and Automotive devrooms, as was announced in this post from some weeks ago. Oh, speaking of the talks, another pretty awesome fact: this year, everything –I mean, every single talk– has been recorded, and videos will be available online soon.

Most of the action, as far as the Xen Project was concerned, happened in the Virtualization & IaaS devroom. Let’s not forget that members of the Xen Project helped record these. A big thank you to them and also a big thank you to the whole FOSDEM video team. Check out out the streams on the FOSDEM website (some are available already). Of course, videos and slides for the Xen talks will be available on the usual aggregators (vimeo and slideshare).


All in all, I personally very much enjoyed having been at FOSDEM. For The Xen  Project, I think we really did good in teaming together with all the members of the community and presenting ourselves to current and prospective users in very good shape. Can’t wait for next year!

Improved Xen support in FreeBSD

FreeBSD Logo
As most FreeBSD users already know, FreeBSD 10 has just been released, and we expect this to be a very good release regarding Xen support. FreeBSD with Xen support includes many improvements, including several performance and stability enhancements that we expect will greatly please and interest users. With many bug fixes already completed, the following description only focuses on new features.

New vector callback

Previous releases of FreeBSD used an IRQ interrupt as the callback mechanism for Xen event channels. While it’s easier to setup, using a IRQ interrupt doesn’t allow to inject events to specific CPUs, basically limiting the use of event channels in disk and network drivers. Also, all interrupts were delivered to a single CPU (CPU#0), not allowing proper interrupt balancing between CPUs.

With the introduction of the vector callback, events can now be delivered to any CPU, allowing FreeBSD to have specific per-CPU interrupts for PV timers and PV IPIs, and balancing the others across the several CPU usually available on a domain.

PV timers

Thanks to the introduction of the vector callback, now we can make use of the Xen PV timer, which is implemented as a per-CPU singleshot timer. This alone doesn’t seem like a great benefit, but it allows FreeBSD to avoid making use of the emulated timers, greatly reducing the emulation overhead and the cost of unnecessary VMEXITs.


As with PV timers, the introduction of the vector callback allows FreeBSD to get rid of the bare metal IPI implementation, and instead route IPIs through event channels. Again, this allows us to get rid of the emulation overhead and unnecessary VMEXITS, providing better performance.

PV disk devices

FLUSH/BARRIER support has been recently added, together with a couple of fixes that allow FreeBSD to run with a CDROM driver under XenServer (which was quite of a pain for XenServer users).

Support for migration

With these new features, migration doesn’t break since it has been reworked to handle the fact that timers and IPIs are also paravirtualized now.

Merge of the XENHVM config into GENERIC

One of the most interesting improvements from a user/admin point of view (and something similar to what the pvops Linux kernel is already doing), the GENERIC kernel on i386 and amd64 now includes full Xen PVHVM support, so there’s no need to recompile a Xen-specific kernel. When run as a Xen guest, the kernel will detect the available Xen features and automatically make use of them in order to obtain the best possible performance.

This work has been done in conjunction between Spectra Logic and Citrix.