Author Archives: Liu Wei

Xen Project 4.6 Planning Opens

With Xen Project 4.5 released in January, we are now one month into 4.6 development window!

My name is Wei Liu and I have been working on various areas in the Xen Project community, including Linux kernel, hypervisor, QEMU and toolstack. Now I’m a co-maintainer of Xen hypervisor’s toolstack and the netback driver in Linux. I was elected release manager for 4.6 release. Thanks everybody for your trust.

I sent an email to xen-devel to kick off a discussion with regard to tweaking the release process for 4.6. The goal is to create smoother developer experience.

My proposed time frame for the Xen 4.6 release is:

  • Development start: 6 Jan 2015
  • Feature freeze: 10 Jul 2015
  • Release date: 9 Oct 2015 (could release earlier)

Below are some slides that explain our release process and how it is changing for Xen 4.6. Get involved and happy hacking!

OSSTest Standalone Mode Step by Step

Xen has long history and many features. Sometimes even experienced developers cannot be sure whether their new code is regression-free. To make sure new code doesn’t cause regression, Ian Jackson developed a test framework called OSSTest. In order to make this framework usable for individual ad-hoc testing, standalone mode was introduced. Recently I played with it and found it useful to share my experience with the community.

Basic Requirements

To run OSSTest in standalone mode, you need to have at least two machines: one is controller, which runs OSSTest itself and various other services, the other is test host, which is used to run test jobs.

For the controller you need to have several services setup:

  • DNS / DHCP, use to translate IP < -> hostname
  • Webserver, provide test box accessible web space, OSSTest will expose configurations / overlays via HTTP and detect test host status via webserver’s access log.
  • PXEboot / Tftp, used to provide Debian installers to test host.
  • NTP, optional

For the test host you need to:

  • enable PXE boot
  • connect to PDU if necessary

Step by step setup

git clone OSSTest tree, master branch, we assume you run everything inside osstest.git directory.

Have a look at README / TODO which contains useful information. You can create a global config file in ~/.xen-osstest/config. In standalone mode you can also create standalone.config in osstest.git directory to override global configurations.

In the configuration you can specify DNS name, name server etc.

HostProp_DhcpWatchMethod leases dhcp3
TftpPath /usr/groups/netboot/

DebianNonfreeFirmware firmware-bnx2
DebianSuite squeeze
DebianPreseed= < <‘END’
d-i clock-setup/ntp-server string

Debian is the de-facto OS in OSSTest, you can find more info in Osstest/ Here we use Squeeze to build binaries and run test jobs, because there’s a bug in Wheezy’s Xen-tools which breaks xen-image-create, which eventually breaks ts-guest-start. Patches to fix that problem have been posted but not yet merged. Hopefully some day in near future we can use Wheezy to run build jobs and test jobs.

Configure test host in OSSTest config file. There can be multiple test hosts in the config file, but I only have one. Here is what I have:

TestHost kodo4
HostProp_kodo4_DIFrontend newt
HostProp_kodo4_PowerMethod xenuse
HostProp_kodo4_Build_Make_Flags -j16
HostFlags_kodo4 need-firmware-deb-firmware-bnx2

There is detailed explanation of what those paramters mean in README. An interesting one is PowerMethod, which in fact points to a module in OSSTest that controls power cycle of the test box. Have a look at Osstest/PDU for all supported modules. Use if your test host is not capable of doing power cycle automatically.

Before actually running OSSTest, you need to make some directories yourself.

  • logs: used to store tarballs from build-* jobs
  • public_html: expose this directory via HTTP server to the test host
  • $TFTPROOT/osstest: OSSTest will write PXE boot information for test hosts here

Make sure test host is able to access contents in public_html, then you can have “WebspaceUrl http://YOUR.CONTROLLER/public_html” in your OSSTest config. Test host will try to fetch all sort of things from there.

Next step will be setting “WebspaceLog /PATCH/TO/WEBSERVER/ACCESS.LOG”. OSSTest watches webserver access log. When test host / guest go to fetch things via HTTP OSSTest gets to know their status. I use Apache2 so I’ve set WebspaceLog to /var/log/apache2/access.log which just works.

Have Debian PXE installers ready. Remember the “DebianSuite” option in your config file? In order to make OSSTest fully functional you will also need to place Debian PXE installers in the right place. You can grab Debian’s PXE installers from any of the Debian archives around the world. And put them under the TFTP you just set up. I would recommend having at least amd64 and i386 in place. Make sure installers are accessible from the test host before proceeding.

By now we’re all set! Next step:


This will reset everything in standalone mode and create standalone.db, which includes test jobs and runtime variables. You can peek what’s inside that database with sqlite3.

The first job to run should be a build-* job. That can: 1) verify your setup is correct; 2) generate a bunch of runtime variables for subsequent test-* jobs.

./sg-run-job build-amd64 # WARNING: this will wipe out your test box

If the job pass, you’re all set. You can play around with other jobs. The default setting of jobs is to wipe out test box on every run. If you don’t want that you need to specify OSSTEST_HOST_REUSE=1 as stated in README.

An example of what I run:

./sg-run-job build-amd64-pvops
OSSTEST_HOST_REUSE=1 ./sg-run-job test-amd64-amd64-xl

If you only want to run a specific testcase, you can try OSSTEST_JOB=$JOBNAME ./ts-XXX host=$TESTHOSTNAME.

Customized tree / revisions

By default OSSTest always fetches trees and revisions from Xenbits. You can easily override them with standalone.config.

Say if I want to test a specific revision of Xen, I have:

export REVISION_XEN=c5e9596cd095e3b96a090002d9e6629a980904eb

in my standalone.confg.

You can look at make-flight to know all the interesting environment variables. (Sorry, no document yet)

Writing new testcases

If you’re interested in writing new testcase, you can do that in two simple steps:

  1. write ts-my-test-case script, you can use any existing testcase as template (they are prefixed with “ts-“)
  2. modify sg-run-job, which has the information for which testcases to run for a specific job

Do have a look as osstest/, in which you can find lots of helpers to accomplish your task.

Writing new test job

The script responsible for making jobs is cs-job-create. When you run OSSTest in standalone mode, it is probably more useful to modify make-flight. You also need to modify sg-run-job to link your new job with testcases.

Hope the above information can help you get started with OSSTest. If you have any problem, do come to Xen-devel and ask.

Some readers may recall the recent announcement of open-sourcing XenRT, and wondering about the relationship between OSSTest and XenRT. Long-term, we expect that XenRT will mature as an open development project and eventually displace OSSTest. But that is not likely to happen for another six months to a year. Developing OSSTest has benefits for the current development, and we hope that getting people involved in writing test cases for OSSTest will be of service in helping people write test cases for XenRT when it becomes available. So we have decided to continue to develop OSSTest until XenRT is ready to replace it.

Have fun! 🙂

Xen network: the future plan

As many of you might have (inevitably) noticed, Xen frontend / backend network drivers in Linux suffered from regression several months back after the XSA-39 fix (various reports here, here and here). Fortunately that’s now fixed (see the most important patch of that series) and the back-porting process to stable kernels is on-going. Now that we’ve put everything back into stable-ish state, it’s time to look into the future to prepare Xen network drivers for the next stage. I mainly work on Linux drivers, but some of the backend improvements ideas should benefit all frontends.

The goal is to improve network performance and scalability without giving up the advanced security feature Xen offers. Just to name a few items:

Split event channels: In the old network drivers there’s only one event channel between frontend and backend. That event channel is used by frontend to do TX notification and RX buffer allocation notification to backend. It is also used by backend to do TX completion and RX notification to frontend. So this is definitely not ideal as TX and RX interferes with each other. So with a little change to the protocol we can split TX and RX notifications into two event channels. This work is now in David Miller’s tree (patch for backend, frontend and document).

Continue reading

Xen Document Day: March 25th, 2013

We have another Xen Document Day coming next Monday, which is March 25th.

Xen Document Days are for people who care about Xen Documentation and want to improve it. Everybody who can and wants contribute is welcome to join!

For a list of items that need work, check out the community maintained TODO and wishlist. We also have a few beginners items in the list. Of course, you can work on anything you like: the list just provides suggestions.

See you on IRC : #xendocs @ freenode !

Improving event channel scalability

As machines are getting more and more powerful today, people want more from the powerful hardware. From a cloud user’s perspective, it is better to run more virtual machines on a single host. Xen currently supports running hundreds of guests on a single physical machine, but we plan to take it to the next magnitude – to run thousands of guests on a single host. To achieve this goal, we need to tackle several bottlenecks, ranging from the hypervisor to the guest kernel to the user space tool stacks. One of the most notable / tricky bit of all the improvements is certainly the event channel scalability, which should enable Dom0 / driver domains to handle more events than before, thus supporting more guests.

What exactly is event channel? In short, event channel is a primitive provided by Xen for event notifications. A typical guest like Linux can map four kinds of events into event channels: Inter-domain notification, IPI (Inter-processor interrupt), VIRQ (virtual IRQ) and PIRQ (physical IRQ).

Xen models devices into a frontend, which runs in the guests, and a backend, which runs in Dom0 or driver domains. The notifications between frontends and backends are done via inter-domain event channels, which means if we run out of event channels we cannot support any more guests. A typical guest equipped with one virtual network interface (or vif) and one virtual disk (or vbd) consumes four event channels in backend domain, that is one for console, one for xenstore, one for vif and one for vbd. We can come up with a rough estimation that a backend domain can support up to (NR_EVENT_CHANNELS_SUPPORTED / 4) guests, which doesn’t even account for the IPIs, VIRQs and PIRQs a backend domain may have. The NR_EVENT_CHANNELS_SUPPORTED for a 64 bit backend domain is 4096 whilst for a 32 bit backend domain is 1024, yielding less than 1024 guests supported by 64 bit backend domain and even less by 32 bit backend domain. So currently, we can support hundreds of guests at best. It gets worse when a guest has several vifs and vbds.

To improve the scalability of event channel, two designs are proposed, one is 3-level event channel which extends the current 2-level mechanism and the other is a ground-up FIFO-based approach. The 3-level design comes with a prototype (Xen patches, Linux patches) while the FIFO-based event channel ABI is currently under discussion on the mailing list (PDF version of draft V2), the final decision has not been made yet. But it is worth mentioning the undergoing work to the public.

The 3-level event channel design takes advantage of the existing 2-level design, simply extending event channel up to next level, which is straightforward. This design supports up to 256K event channels for a 64 bit backend domain and 32K event channels for a 32 bit backend domain, which in practice is more than enough in the long run. The downsides of this design is that:

  1. it uses global mapping address, which is only 1G
  2. a flat bitmap of events provides no priorities

Solving problem 1 is straightforward. DomU will never use so many event channels, so we just enable 3-level event channel for Dom0 and driver domains. Problem 2 is by design not easily solvable.

The FIFO-based design starts from scratch, which has following advantages over the 3-level design:

  1. event priorities
  2. use FIFO to ensure events fairness
  3. flexible / configurable event channel number for individual guest to reduce memory footprint
  4. consistent ABI across 32 / 64 bit guest / hypervisor

The main downside for FIFO-based design is, well, it is not yet implemented. Certainly many details need to be pinned down (such as avoid using too much global mapping address) and benchmarks need to be conducted. Event handling code is complex by natural and we should be patient to wait for this design to become mature.

From a user’s point of view, either of the two designs should improve event channel scalablity and take Xen’s ability to support guests to next magnitude. 3-level design is proposed to meet the time frame of 4.3 release because of its simplicity. The FIFO-based design is proposed since we are defining a new ABI and our developers think it is high time we start from scratch. The door is still open, feel free to share your views, concerns and suggestions on Xen development list.

My Journey with Xen

Hi everyone!

I’m Wei Liu, a graduate student who is pursuing his master’s degree from China. If you read posts on from time to time, you might remember me. I was participant of Google Summer of Code 2011 and worked on “Virtio on Xen” project for It turned out that I managed to contribute my humble two cents to open source Xen during GSoC, so Citrix (the backing company of offered me an internship position in Cambridge, UK.

The internship is almost over, so I thought it would be nice to share my experience of GSoC, working for Xen platform team, what I’ve learned and what I’ve accomplished so far.

Talking about GSoC now seems weird, but the point is I never wrote about it last year after I finished my project, so it is worth mentioning it now. Further more, GSoC is the starting point when I started to seriously work for open source Xen, it will make a good beginning of my story.

Continue reading