Monthly Archives: June 2015

Project Raisin – Raise Xen!

It all started with pvgrub2: it was March 2015 and I wanted to add grub2 to the Xen build system. We were already building grub-legacy as part of the Xen build, so that we could produce a pvgrub binary to be used to boot PV guests. After Vladimir ‘phcoder‘ Serbinenko’s good work on grub2, the latest and greatest upstream grub2 could be built with Xen support and used to boot PV guests. It made perfect sense to add grub2 to the Xen build system too, right? Maybe not. Unexpectedly some key Xen Project contributors pushed back: “there doesn’t seem to be a good reason for cloning and building yet another third-party project as part of the Xen build”, wrote David Vrabel.

Conflicting requirements

It was then that I realized that we have two contrasting set of requirements: on one hand we want to support users that clone xen-unstable, build everything from source, and expect the system to be fully ready after typing ./configure; make; make install. On the other hand, we also want to support distros and product groups that take Xen releases and integrate them into their Linux distros or enterprise build systems. The former want things like grub2 to be part of the xen-unstable build, because the grub2 package provided by their distro doesn’t necessarily comes with Xen support enabled. While the distro packagers are already building a grub2 package and certainly don’t want xen-unstable to go and clone grub2 again. They probably abhor the whole idea of xen-unstable git cloning external trees without their explicit assent. In fact they had been carrying patches to make sure xen-unstable doesn’t clone anything else “behind their back”, until we provided build options to disable all the third-party builds.

Raisin: Xen’s DevStack

How to find a solution that would make both camps happy? Surely others must have had the same issue. Is there another open source project that has to build several separate components in order to be fully functional? Yes, of course, there are many. One of them is OpenStack and it solves the problem by providing a set of scripts called DevStack, which build and setup the system from scratch.

This is where “Raisin” comes from. I announced the new project on the 31st of March 2015. The idea is that Raisin takes care of building Xen and all the other components, which are required to have a fully functional Xen system, but that don’t belong to xen-unstable. For example QEMU, SeaBIOS, and, of course, grub2. Users that build everything from source will clone Raisin to find a single place where they can build all the latest and greatest Xen stuff with a single command. Raisin can be very useful to setup a development environment too. Distro people can refer to Raisin as the most common way to build, install and configure Xen and related components, but they are unlikely to actually use it to build their packages. Raisin helps Xen developers improve the boundaries and interfaces between Xen and external components, by making such boundaries clearer and more explicit. Things like QEMU and SeaBIOS, currently cloned and built by xen-unstable, will be moved out to Raisin, making both Xen maintainers and distro packagers happier. Other Xen related components, that are good to have but not actually required, such as libvirt, will find their place in Raisin too.

Raisin: where we are, what’s next

After few busy months of development, we now have a set of bash scripts that can install dependencies and build Xen, QEMU, qemu-traditional, SeaBIOS, OVMF, Grub2, Libvirt and Linux with a single command. All you need to do is edit the config file, type raise -y build, go get a coffee, and everything will be ready when you come back. Raisin is not tied to a specific version of Xen. In fact, one can choose any git tags or commit ids newer than Xen 4.5 (RELEASE-4.5.0 is the git tag for the Xen 4.5 release) and Raisin will build it. Other commands are available to install and configure the system with the most common settings. Give a look at the README for an introduction on how to use the command line tool.

During the last few weeks I have been working on integrating Raisin in OSSTest, the automated testing framework run by Xen Project. I am currently adding a new test to validate Raisin itself, but going forward it makes sense to actually use Raisin to build Xen, QEMU and anything else OSSTest needs, similarly to what DevStack does for the OpenStack gate.

Making testing easier and accessible to everybody

Talking about tests, this is another area where Raisin can help greatly. I always liked the idea of providing a set of unit and functional tests, quick and simple to run, that can be executed by any Xen contributors to validate their changes before sending a patch to xen-devel. However we didn’t really have place to put them. OSSTest is too big and tightly coupled to the Xen Project Test Lab infrastructure for this use case, and the last thing xen-unstable needs is more scripts. On the other hand, Raisin is at the right abstraction level to run functional tests for the components it already builds. I introduced a few simple tests, which can stack on top of each other, to create busybox based PV and HVM guests. I plan to continue adding more tests, then expose them to OSSTest via Raisin, so that they will be continuously run by the Xen Project Test Lab. But, at the same time, anybody can still manually execute them on their test box with a single raise test command. I am hoping to be able to start asking contributors to run Raisin tests before submitting patches early in the next release cycle. If you use Xen and know bash scripting, you should consider writing a Raisin test to validate your favourite functionality today!

Raisin, you didn’t know you needed it, you can’t live without it ;-)

Resources

The Raisin git repository is available here. The README is up to date and describes the command line interface. We also have quickstart guide on our wiki. Raisin patches are discussed on xen-devel and follow the regular Xen development process.

Xen Project 4.5.1 Maintenance Release Available

I am pleased to announce the release of Xen 4.5.1. We recommend that all users of the 4.5 stable series update to this first point release.

Xen 4.5.1 is available immediately from its git repository:

    xenbits.xenproject.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.5
    (tag RELEASE-4.5.1)

or from the Xen Project download page at www.xenproject.org/downloads/xen-archives/xen-45-series/xen-451.html.

This release contains many bug fixes and improvements. For example:

  • Removal of race conditions in the Xen default toolstack that affected libvirt, in particular when used with OpenStack (this release contains all changes that we use in the Xen Project OpenStack CI loop; also see related OpenStack news);
  • Stability improvements to CPUPOOL handling, in particular when used with different schedulers;
  • Stability improvements to EFI support on some x86 platforms;
  • Stability improvement to handling of nested virtualisation on x86;
  • Various improvements to 32 and 64 bit ARM support;
  • Various improvements to better integrate and support rump kernels;
  • Error handling improvements; and
  • Security fixes since the release of Xen 4.5.0,

For a complete list of changes in this release, please check lists of changes on the download page.

2015 Xen Project Developer Summit Line-up Announced

I am pleased to announce the schedule for the Xen Project Developer Summit. The event will take place in Seattle on August 17-18, 2015.

The Xen Project Developer Summit brings together its community of developers and power users. Each year the event features the latest developments, best practices, collaboration, product roadmap updates and future planning from developers who are leading the way in server density, million-node data centers, automotive, mobile, graphic-intensive workloads, cloud and enterprise security.

For the first time, Xen Project Developer Summit and KVM Forum will co-host a Hackathon on Aug. 18 aimed at fostering technical collaboration between the two leading open source hypervisors in IT today. KVM and Xen users and developers will have the opportunity to collaborate and delve into work on libvirt code. A co-hosted evening event will be held that night.

Following is a sampling of confirmed speakers and presentations to be discussed in Seattle:

  • Dario Faggioli, senior software engineer, Citrix, and Meng Xu, PhD Student, University of Pennsylvania, will co-present the state of scheduling in Xen and the hypervisor provides a set of schedulers, each one suited for specific use cases.
  • Sainath Grandhi, Core OS engineer, Intel will present Xen containers to run Docker container applications sandboxed in a small VM as an alternative to bare-metal containers, providing tighter security and resource isolation.
  • Julien Grall, software engineer, Citrix, will target developers making a product based on Xen ARM requiring device assignment.
  • Juergen Gross, Linux kernel developer, SUSE, will outline suggestions for a systems architecture that is much more fault tolerant against hardware and software failures.
  • Manish Jaggi, Xen/KVM hypervisor technical lead, Cavium, will talk about Xen support for ThunderX, a family of highly integrated, multi-core SoC processors based on 64-bit ARMv8 architecture, for data center and cloud applications.
  • Liu Jinsong, PM and RAS maintainer, Alibaba, will introduce live migration at AliCloud, summarizing the technical virtualization, network, and storage problems that need to be solved to make live migration work at AliCloud.
  • Mark Kraeling, product manager, GE Transportation, will talk about virtualizing the locomotive and how GE uses about how GE uses Xen for x86-based processors, and KVM for ARM-based processors.
  • Tamas Lengyel, security researcher at Novetta, University of Connecticut PhD student, will shed more light on current trends in virtualization security, pitfalls and new features coming in 4.6.
  • Wei Liu, Xen 4.6 release manager, Citrix, will give a status report on the upcoming Xen Project 4.6 release.
    Stefano Stabellini, senior principle software engineer, Citrix, will explain how to deploy OpenStack using the Xen Project hypervisor to run your VMs.
  • Zhi Wang, engineer, Intel, will provide a detailed update on the evolution of Intel Graphics Virtualization Technology for full GPU virtualization.
  • Konrad Wilk, software director, Oracle, will discuss the design and functionality of xSplice, which offers a method for live patching without requiring a system to reboot.
  • Marc Zyngier, kernel hacker, ARM, will share a hypervisor agnostic view of virtualization extensions added to the latest ARM architecture.

Birds of a Feather session and Discussions

Besides presentations, the developer summit will also provide an opportunity for in-depth interactive discussions (Birds of a Feather sessions), which allow deep interaction and collaboration between Xen Project developers and community members. These will happen in a second track alongside part of the main event. To submit a BoF, please send an email to community dot manager at xenproject dot org and chose a BoF slot, title and short description or fill out this wiki page.

Registration

For more information about Xen Project Developer Summit 2015, including how to register and to view the complete schedule, visit: events.linuxfoundation.org/events/xen-project-developer-summit.