Tag Archives: development

Project Raisin – Raise Xen!

It all started with pvgrub2: it was March 2015 and I wanted to add grub2 to the Xen build system. We were already building grub-legacy as part of the Xen build, so that we could produce a pvgrub binary to be used to boot PV guests. After Vladimir ‘phcoder‘ Serbinenko’s good work on grub2, the latest and greatest upstream grub2 could be built with Xen support and used to boot PV guests. It made perfect sense to add grub2 to the Xen build system too, right? Maybe not. Unexpectedly some key Xen Project contributors pushed back: “there doesn’t seem to be a good reason for cloning and building yet another third-party project as part of the Xen build”, wrote David Vrabel.

Conflicting requirements

It was then that I realized that we have two contrasting set of requirements: on one hand we want to support users that clone xen-unstable, build everything from source, and expect the system to be fully ready after typing ./configure; make; make install. On the other hand, we also want to support distros and product groups that take Xen releases and integrate them into their Linux distros or enterprise build systems. The former want things like grub2 to be part of the xen-unstable build, because the grub2 package provided by their distro doesn’t necessarily comes with Xen support enabled. While the distro packagers are already building a grub2 package and certainly don’t want xen-unstable to go and clone grub2 again. They probably abhor the whole idea of xen-unstable git cloning external trees without their explicit assent. In fact they had been carrying patches to make sure xen-unstable doesn’t clone anything else “behind their back”, until we provided build options to disable all the third-party builds.

Raisin: Xen’s DevStack

How to find a solution that would make both camps happy? Surely others must have had the same issue. Is there another open source project that has to build several separate components in order to be fully functional? Yes, of course, there are many. One of them is OpenStack and it solves the problem by providing a set of scripts called DevStack, which build and setup the system from scratch.

This is where “Raisin” comes from. I announced the new project on the 31st of March 2015. The idea is that Raisin takes care of building Xen and all the other components, which are required to have a fully functional Xen system, but that don’t belong to xen-unstable. For example QEMU, SeaBIOS, and, of course, grub2. Users that build everything from source will clone Raisin to find a single place where they can build all the latest and greatest Xen stuff with a single command. Raisin can be very useful to setup a development environment too. Distro people can refer to Raisin as the most common way to build, install and configure Xen and related components, but they are unlikely to actually use it to build their packages. Raisin helps Xen developers improve the boundaries and interfaces between Xen and external components, by making such boundaries clearer and more explicit. Things like QEMU and SeaBIOS, currently cloned and built by xen-unstable, will be moved out to Raisin, making both Xen maintainers and distro packagers happier. Other Xen related components, that are good to have but not actually required, such as libvirt, will find their place in Raisin too.

Raisin: where we are, what’s next

After few busy months of development, we now have a set of bash scripts that can install dependencies and build Xen, QEMU, qemu-traditional, SeaBIOS, OVMF, Grub2, Libvirt and Linux with a single command. All you need to do is edit the config file, type raise -y build, go get a coffee, and everything will be ready when you come back. Raisin is not tied to a specific version of Xen. In fact, one can choose any git tags or commit ids newer than Xen 4.5 (RELEASE-4.5.0 is the git tag for the Xen 4.5 release) and Raisin will build it. Other commands are available to install and configure the system with the most common settings. Give a look at the README for an introduction on how to use the command line tool.

During the last few weeks I have been working on integrating Raisin in OSSTest, the automated testing framework run by Xen Project. I am currently adding a new test to validate Raisin itself, but going forward it makes sense to actually use Raisin to build Xen, QEMU and anything else OSSTest needs, similarly to what DevStack does for the OpenStack gate.

Making testing easier and accessible to everybody

Talking about tests, this is another area where Raisin can help greatly. I always liked the idea of providing a set of unit and functional tests, quick and simple to run, that can be executed by any Xen contributors to validate their changes before sending a patch to xen-devel. However we didn’t really have place to put them. OSSTest is too big and tightly coupled to the Xen Project Test Lab infrastructure for this use case, and the last thing xen-unstable needs is more scripts. On the other hand, Raisin is at the right abstraction level to run functional tests for the components it already builds. I introduced a few simple tests, which can stack on top of each other, to create busybox based PV and HVM guests. I plan to continue adding more tests, then expose them to OSSTest via Raisin, so that they will be continuously run by the Xen Project Test Lab. But, at the same time, anybody can still manually execute them on their test box with a single raise test command. I am hoping to be able to start asking contributors to run Raisin tests before submitting patches early in the next release cycle. If you use Xen and know bash scripting, you should consider writing a Raisin test to validate your favourite functionality today!

Raisin, you didn’t know you needed it, you can’t live without it ;-)

Resources

The Raisin git repository is available here. The README is up to date and describes the command line interface. We also have quickstart guide on our wiki. Raisin patches are discussed on xen-devel and follow the regular Xen development process.

October 29: Xen Project 4.5 RC1 Test Day

TestDay

The Xen Project team is pleased to announce the first Test Day for 4.5 Release Candidate 1 will be held on October 29, 2014.  The 4.5 release is just a few weeks away, so this is an important event in our development calendar.

Test Days insure that the upcoming release is ready for production.  It also allows power users to test out the upcoming release in their own environment.

Subsequent Test Days are expected to be scheduled roughly ever other week until it is determined that the software is ready for release.

General Information about Test Days can be found here:
http://wiki.xenproject.org/wiki/Xen_Project_Test_Days

and specific instructions for this Test Day are located here:
http://wiki.xenproject.org/wiki/Xen_4.5_RC1_test_instructions

XEN 4.5 FEATURE DEVELOPERS:

If you have a new feature which is cooked and ready for testing in RC1, we need to know about it and how to test it. Either edit the instructions page or send me a few lines describing the feature and how it should be tested.

Currently, this Test Day is focused on general tests (e.g., “Does the software compile, install, and do the things it normally does, regardless of hardware platform?”).  If you have specific new functionality which needs testing in RC1, we need to know about it and how to test it.

EVERYONE:

Please join us on Wednesday, October 29, and help make sure the next release of Xen Project software is the best one yet!  To make room for the Test Day, we’ve moved Document Day back to November 5, so join us then as we improve documentation with a special theme of integration this month.

The Windows PV Drivers Sub-Project

by Paul Durrant

Back in 2013 Citrix made XenServer fully open source. As part of that work the previously closed Windows drivers for paravirtual devices were opened up and made available to the community on GitHub. These drivers were still very much tied to XenServer though because of assumptions that were made about the platform and reliance on certain patches carried in the XenServer patch queues.

Shortly after setting up the driver repositories on GitHub I started removing these assumptions and dependencies on the XenServer platform and produced a set of drivers that could be used on most Xen installations. This put the code in a good state to approach the project community and Xen Project Advisory Board with a proposal for an incubation sub-project. I’m happy to say that this was and we are well under way in getting things set up. There is a new with information on the driver source repositories (which are now hosted on xenbits under a new pvdrivers/win sub-directory), instructions on how to build and install the drivers, and guidelines on how to contribute to the project.

Overview

The PV drivers are split into five packages:

XenBus

This is the key package that supports all other PV drivers. It provides the XENBUS driver which binds to either the XenServer variant of the (see in the hypervisor source repository) or the ubiquitous Xen Platform PCI Device, both of which are provided to HVM guests by QEMU.
This driver establishes communication with the Xen hypervisor and enumerates the paravirtual classes specified by the toolstack in xenstore. It also provides APIs to core Xen functionality such as event channels and the grant table.

XenVif

This package contains the network class driver XENVIF. This driver makes use of the APIs provided by XENBUS to implement the PV network protocol. It enumerates PV network devices (specified in the guest area in xenstore under device/vif) and provides a simple API to use the protocol.

XenNet

This package contains the NDIS 6 network device driver XENNET. This driver binds to the devices enumerated by XENVIF and provides a relatively thin glue layer between the Microsoft-defined NDIS miniport interface and the PV network API exported by XENVIF.

XenVbd

This package contains the storage class driver XENVBD and associated crash-kernel driver XENCRSH. XENVBD makes use of the APIs provided by XENBUS and the Windows STORPORT miniport interface to provide limited SCSI HBA functionality to the OS. It is limited in the fact that it only supports the operations necessary for the Windows generic DISK driver to function on the devices exposed by the HBA.

XenIface

This package contains the XENIFACE driver which creates WMI objects providing access to xenstore (see ) and system time, and a minimal user-space guest agent which uses those objects to provide facilities to cleanly shut down or reboot a guest and also re-synchronize guest time after a VM migrate/restore.

Xen Summit Boston Speakers Wanted

Attention, Attention, Attention. The Xen Summit Program Committee for the Boston event in June (23 – 24) is busy reviewing submitted topics and is waiting for your proposal. We are actively putting together the agenda and would like nothing more than to have you as a speaker. This is your chance to promote yourself and your work to the Xen.org community. To assist us, we are asking all interested speakers to submit their topics early this year so we can create the best possible event without having any last minute submissions which may get missed as many Xen Program Committee members will be taking vacation early this summer. Please submit your topics to stephen.spector@xen.org and I will ensure the Program Committee receives it for a timely review.

Call for Topics for Xen Summit

Calling all speakers, calling all speakers… The Xen Summit 2008 (June 23 – 24 in Boston, MA) agenda being created by the Xen Summit Program Committee is ready for development and we need your help. Have you had a desire to tell others about the great work you have done with Xen or are looking for fame and fortune in the Xen community? If so, we are looking for you. Please submit your topic ideas for consideration to stephen.spector@citrix.com by April 30, 2008 and I will ensure that it is considered by the Xen Summit Program Committee. The Xen Summit Program Committee will then review your idea and request further information such as an abstract, etc. All speakers chosen for Xen Summit 2008 will get all the glory and fame necessary to continue on as a leader in the Xen community.