Author Archives: Stefano Stabellini

SWIOTLB by Morpheus

The following monologue explains how Linux drivers are able to program a device when running in a Xen virtual machine on ARM.

The problem that needs to be solved is that Xen on ARM guests run with second stage translation in hardware enabled. That means that what the Linux kernel sees as a physical address doesn’t actually correspond to a machine address. An additional translation layer is set by the hypervisor to do the conversion in hardware.

Many devices use DMA to read or write buffers in main memory and they need to be programmed with the addresses of the buffers. In the absence of an IOMMU, DMA requests don’t go through the same physical to machine translation set by the hypervisor for virtual machines, devices need to be programmed with machine addresses rather than physical addresses. Hence the problem we are trying to solve.

Definitions of some of the technical terms used in this article are available at the bottom of the page.

Given the complexity of the topic, we decided to ask for help to somebody with hands-on experience with teaching the recognition of the differences between “virtual” and “real”.


At last.
Please. Come. Sit.
Xen Project Matrix

Do you realize that everything running on Xen is a virtual machine — that Dom0, the OS from which you control the rest of the system, is just the first virtual machine created by the hypervisor? Usually Xen assigns all the devices on the platform to Dom0, which runs the drivers for them.

I imagine, right now, you must be feeling a bit like Alice, tumbling down the rabbit hole?
Let me tell you why you are here.

You are here because you want to know how to program a device in Linux on Xen on ARM.

Continue reading

Bringing Open Source Communities closer together

For the last several years, the Xen developer community has been increasing its ability to collaborate well with other projects. We succeeded in finally getting the necessary infrastructure for dom0 support into Linux in 2011. We have upstreamed the most important changes to QEMU, and will be using an upstream QEMU based tree in the Xen 4.3 release. Additionally, during the the last several months we have improved Xen support in libvirt. We are not just looking to upstream projects but we are also improving our relationship with downstream users, like Linux distros. We worked closely with Debian, Ubuntu and CentOS, for example announcing the availability of Xen packages for CentOS 6 at FOSDEM few months ago.

Today is the perfect day to announce that in addition to these efforts, we have been working behind the scenes on some other exciting new initiatives designed to increase our collaboration and influence with other projects.

Continue reading

Xen @ Linaro Connect Asia 2013

Two weeks ago today was the first day of Linaro Connect Asia: the event was held at the Gold Coast Hotel in Hong Kong and was similar in format and attendance to the previous one in Copenhagen. All the major players in the ARM world came together to speak about the future of the industry and build an healthy Open Source ecosystem for ARM.

This time Ian and I didn’t just listen: we came to present Xen on ARM and to demonstrate the project on stage (a video is available here, but unfortunatelty it is focused on the audience rather than the speakers). Our talk was on Monday, the first day of the conference, and was very well attended: the room was completely full, people were standing and sitting in the corridor to listen to us! Since the talk was Xen specific and rather technical, I am surprised that we managed to raise so much interest. Not even good talks at XenSummit have so many attendees. It is also an indication of how active and vibrant the Linaro community is, managing to get together so many kernel engineers in one room.

We explained the basic ideas behind the original Xen architecture and showed the benefit of some of the more advanced configurations, like disaggregating dom0 and running multiple driver domains. For example one could have a network driver domain with the network card assigned to it, running the network backend. The same can be done for the disk, assigning the SATA controller to a disk driver domain, running the block backend. Driver domains can be restarted independently, without compromising the stability of the system. They improve scalability because they remove some of the load from Dom0. Most importantly, given that they don’t need to be privileged, they improve security and isolation. As a matter of fact, one of the most security oriented operating systems out there, QubesOS, uses them extensively. The ARM port of Xen retains these architectural advantages and renovates the overall design by having a single hypercall interface for 32-bit and 64-bit virtual machines and by removing the distinction between PV and HVM guests. The new kind of guest supported by Xen on ARM provides the advantages of both, without any of the disadvantages: it uses PV interfaces for IO early at boot, it doesn’t require any emulation (no QEMU) and exploits the hardware as much as possible. Xen is running entirely within EL2, or hypervisor mode and it uses second stage translation in the MMU to assign memory to virtual machines. Xen exploits generic timers to export a virtual timer interface to guests and uses the GIC to inject physical and virtual interrupts into virtual machines. Guest operating systems use HVC to issue hypercalls. No ARM virtualization features are left unused.

After the presentation we showed Xen running on the Arndale board and Xen 64-bit running on the ARMv8 emulator: both demos went smoothly and the crowd enjoyed watching us playing tetris on a virtual machine on Xen on ARM.

That was Monday and the activities around Xen were just starting: Mark Heath (XenServer VP of engineering) and Lars Kurth ( Community manager) gave the keynote the following morning (the slides are available here). They talked about the reasons why Xen is important for Citrix and why Xen and ARM make a very good match. Given how significant is this project for Citrix, Mark exploited the opportunity to announce that Citrix is going to join the Linaro Enterprise Group (LEG) as soon as possible.

A couple of days of meetings with the LEG members and productive hacking sessions in the afternoon followed. Ian managed to boot the first 64-bit Linux guest on a 64-bit hypervisor during that time. The last day reserved a couple of hours for a demo session: we had a Xen booth with the two demos we showed on Monday, plus Xen running natively on the Samsung Chromebook as an added bonus. Many people stopped by to watch the demos, ask questions about Xen, and see it running first hand.

Overall it was a very productive week: I don’t think that anybody in the ARM world could dismiss Xen on ARM as vapoware or uninteresting after Linaro Connect Asia 2013. It’s happening and it’s happening now.

News from Linaro Connect

A couple of weeks ago I went to Copenhagen to attend Linaro Connect and Ubuntu Developer Summit for the first time. I was really impressed by the size of the conference, I wasn’t expecting so many people, it certainly rivals LinuxCon in terms of attendance.

All the best minds in the ARM world together in the same hotel for a week: the list of attendees includes Arnd Bergmann (Linaro), Olof Johansson (Google), Grant Likely (Linaro), David Rusling (Linaro), Jon Masters (Red Hat) and many others. You can imagine the level of technical discussions that was going on.

Continue reading

Xen ARM in Linux!

Last weekend Linus Torvalds pulled the Xen on ARM patches in his Linux tree, so as of Saturday the 7th of October, we have Xen ARM in upstream Linux!

This makes Xen the first hypervisor supported by Linux on the ARM platform!

Working on ARM has been a very pleasant experience for me: the documentation of the hardware is well written and complete, the virtualization extensions are useful and fit our architecture very well, the ARM emulator comes with a nice debugger that helped me figure out some of the most difficult problems I had.

But beyond the hardware and the development tools, I was really impressed by how welcoming the Linux ARM Community has been to me: from the invite to the ARM session at the Kernel Summit, to the guidance through the upstreaming process and in general the feedback I received to my work. Nowadays, the Linux Community is usually perceived as not being friendly to newcomers, but it hasn’t certainly been the case for me! In particular I would like to thanks Arnd Bergmann, Marc Zyngier, and the Linaro folks. It has been great working with you. I would also like to thank Konrad Rzeszutek Wilk because he went out of his way to help me upstream my ARM work: I couldn’t have done it without him.

As the Xen on ARM patches were pulled by Linus, I was also appointed as Xen ARM maintainer in the Linux kernel. This is a new responsibility for me and I am not going to take it lightly. I am looking forward to work closely with Arnd, Russell, Konrad and the other Linux maintainers to make Linux the best operating system to run virtual machines and inside virtual machines on Xen on ARM!

What’s next

While Xen on ARM in Linux is certainly a major milestone, there are still a lot of things to do. Right now I am busy trying to run Xen on a Cortex A15 Versatile Express development platform while Ian Campbell already started the ARMv8 Xen port!
Stay tuned if you want to run Xen 64-bit on ARM.

See Also

If you are interested in the Xen on ARM project, you might want to read the slides of the presentation I gave at XenSummit 2012:

Xen @ LinuxCon 2011

Konrad and I have been representing Xen at KVM Forum and LinuxCon 2011 last week. First of all I want to thanks Alexander Graf, Anthony Liguori and all the other members of the KVM community for the warm welcome and fruitful collaboration on many hot topics about virtualization. KVM and Xen have Linux and Qemu in common and share many problems that we can help each other solve. We had very interesting discussions regarding VFIO, USB redirection over the network, AHCI emulation, SPICE and many other subjects.

Then LinuxCon came along with many attendees and interesting presentations on the newest challenges in storage, network, ARM ports and so on. I particularly liked What to Expect From Linux Storage by James Bottomley and Optimizing Performance of Persistent Data Storage in KVM-based Clouds by Khoa Huynh. One thing that I noticed is that people really understand now that Xen is an integral part of the Linux ecosystem. Thanks to Linux 3.0 containing all the Xen pieces needed to run Dom0, everybody knows that we are first class citizens in the Linux world.

Somebody asked a question to Linus Torvalds about the supposed war the is going on between Xen and KVM, that lead to the famous quote “virtualization is evil“. The truth is that the is no war, I don’t know why people think that there is one. Xen and KVM might be competitors, but are different projects, with different architectures and constraints, both Open Source and both GPLv2, that have many goals and components in common. Competing and collaborating at the same time is in the very nature of Open Source and continuously happens at very different levels, between Gnome and KDE, among Linux distributions, among hardware vendors, and so on. There is really nothing strange or new going on.

The good news for the Xen project is that even though Linus dislikes virtualization technologies (they are evil after all :-)), he stated that he is pleased with the way Xen is run nowadays.
Technology discussions aside, Vancouver is a really beautiful city and the LinuxCon Gala “Roaring 20’s” was one of the best events I have ever attended. I recommend to checkout the pictures on the Linux Foundation website as soon as they are available 🙂