Tag Archives: policy

RT-Xen: Real-Time Virtualization in Xen

RT-XenThe researchers at Washington University in St. Louis and University of Pennsylvania are pleased to announce, here on this blog, the release of a new and greatly improved version of the RT-Xen project. Recent years have seen increasing demand for supporting real-time systems in virtualized environments (for example, the Xen-ARM projects and several other real-time enhancements to Xen), as virtualization enables greater flexibility and reduces cost, weight and energy by breaking the correspondence between logical systems and physical systems. As an example of this, check out the video below from the 2013 Xen Project Developer Summit

The video describes how Xen could be used in an in-vehicle infotainement system.

In order to combine real-time and virtualization, a formally defined real-time scheduler at the hypervisor level is needed to provide timing guarantees to the guest virtual machines. RT-Xen bridges the gap between real-time scheduling theory and the virtualization technology by providing a suite of multi-core real-time schedulers to deliver real-time performance to domains running on the Xen hypervisor.

Background: Scheduling in Xen

In Xen, each domain’s core is abstracted as a Virtual CPU (VCPU), and the hypervisor scheduler is responsible for scheduling VCPUs. For example, the default credit scheduler would assign a weight per domain, which decides the proportional share of CPU cycles that a domain would get. The credit scheduler works great for general purpose computing, but is not suitable for real-time applications due to the following reasons:

  1. There is no reservation with credit scheduler. For example, when two VCPUs runs on a 2 GHz physical core, each would get 1 GHz. However, if another VCPU also boots on the same PCPU, the resource share shrinks to 0.66 GHz. The system manager have to carefully configure the number of VMs/VCPUs to ensure that each domain get an appropriate amount of CPU resource;
  2. There is little timing predictability or real-time performance provided to the VM. If a VM is running some real-time workload (video decoding, voice processing, and feedback control loops) which are periodically triggered and have a timing requirement — for example, the VM must be scheduled every 10 ms to process the data — there is no way the VM can express this information to the underlying VMM scheduler. The existing SEDF scheduler can help with this, but it has poor support for multi-core.

RT-Xen: Combining real-time and virtualization

RT-Xen aims to solve this problem by providing a suite of real-time schedulers. The users can specify (budget, period, CPU mask) for each VCPU individually. The budget represents the maximum CPU resource a VCPU will get during a period; the period represents the timing quantum of the CPU resources provided to the VCPU; the CPU mask defines a subset of physical cores a VCPU is allowed to run. For each VCPU, the budget is reset at each starting point of the period (all in milliseconds), consumed when the VCPU is executing, and deferred when the VCPU has budget but no work to do.

Within each scheduler, the users can switch between different priority schemes: earliest deadline first (EDF), where VCPU with earlier deadline has higher priority; or rate monotonic (RM), where VCPU with shorter period has higher priority. As a results, not only the VCPU gets a resource reservation (budget/period), but also an explicit timing information for the CPU resources (period). The real-time schedulers in RT-Xen delivers the desired real-time performance to the VMs based on the resource reservations.

To be more specific, the two multi-core schedulers in RT-Xen are:

  • RT-global: which uses a global run queue to hold all VCPUs (in runnable state). It is CPU mask aware, and provides better resource utilization, as VCPU can migrate freely between physical cores (within CPU mask)
  • RT-partition: which uses a run queue per physical CPU. In this way, each physical CPU only looks at its own run queue to make scheduling decisions, which incurs less overhead and potentially better cache performance. However, load-balancing between physical cores is not provided in the current release.

Source Code and References

The developers of RT-Xen are looking closely at how to integrate both schedulers into the Xen mainstream. In the meantime, please check out publications at [EMSOFT’14], [EMSOFT’11], [RTAS’12] and source code.

Fedora 20 Virtualization Test Day is today!

Fedora Logo

Yes, today (Tuesday, October 8th) is one of the Fedora 20 Test Days, more specifically, Virtualization Test Day.

Specific information regarding testing Xen on the new Fedora can be found in this Wiki page. For attending and participating, join us now on IRC at #fedora-test-day (Freenode)!

Fedora 20 will be one of the first mainstream distros shipping Xen 4.3, so come and help us making sure it will work great for you and all Fedora users!!

OSSTest Standalone Mode Step by Step

Xen has long history and many features. Sometimes even experienced developers cannot be sure whether their new code is regression-free. To make sure new code doesn’t cause regression, Ian Jackson developed a test framework called OSSTest. In order to make this framework usable for individual ad-hoc testing, standalone mode was introduced. Recently I played with it and found it useful to share my experience with the community.

Basic Requirements

To run OSSTest in standalone mode, you need to have at least two machines: one is controller, which runs OSSTest itself and various other services, the other is test host, which is used to run test jobs.

For the controller you need to have several services setup:

  • DNS / DHCP, use to translate IP < -> hostname
  • Webserver, provide test box accessible web space, OSSTest will expose configurations / overlays via HTTP and detect test host status via webserver’s access log.
  • PXEboot / Tftp, used to provide Debian installers to test host.
  • NTP, optional

For the test host you need to:

  • enable PXE boot
  • connect to PDU if necessary

Step by step setup

git clone OSSTest tree, master branch, we assume you run everything inside osstest.git directory.

Have a look at README / TODO which contains useful information. You can create a global config file in ~/.xen-osstest/config. In standalone mode you can also create standalone.config in osstest.git directory to override global configurations.

In the configuration you can specify DNS name, name server etc.

DnsDomain uk.xensource.com
HostProp_DhcpWatchMethod leases dhcp3 dhcp.uk.xensource.com:5556
TftpPath /usr/groups/netboot/

DebianNonfreeFirmware firmware-bnx2
DebianSuite squeeze
DebianMirrorHost debian.uk.xensource.com
DebianPreseed= < <‘END’
d-i clock-setup/ntp-server string ntp.uk.xensource.com

Debian is the de-facto OS in OSSTest, you can find more info in Osstest/Debian.pm. Here we use Squeeze to build binaries and run test jobs, because there’s a bug in Wheezy’s Xen-tools which breaks xen-image-create, which eventually breaks ts-guest-start. Patches to fix that problem have been posted but not yet merged. Hopefully some day in near future we can use Wheezy to run build jobs and test jobs.

Configure test host in OSSTest config file. There can be multiple test hosts in the config file, but I only have one. Here is what I have:

TestHost kodo4
HostProp_kodo4_DIFrontend newt
HostProp_kodo4_PowerMethod xenuse
HostProp_kodo4_Build_Make_Flags -j16
HostFlags_kodo4 need-firmware-deb-firmware-bnx2

There is detailed explanation of what those paramters mean in README. An interesting one is PowerMethod, which in fact points to a module in OSSTest that controls power cycle of the test box. Have a look at Osstest/PDU for all supported modules. Use manual.pm if your test host is not capable of doing power cycle automatically.

Before actually running OSSTest, you need to make some directories yourself.

  • logs: used to store tarballs from build-* jobs
  • public_html: expose this directory via HTTP server to the test host
  • $TFTPROOT/osstest: OSSTest will write PXE boot information for test hosts here

Make sure test host is able to access contents in public_html, then you can have “WebspaceUrl http://YOUR.CONTROLLER/public_html” in your OSSTest config. Test host will try to fetch all sort of things from there.

Next step will be setting “WebspaceLog /PATCH/TO/WEBSERVER/ACCESS.LOG”. OSSTest watches webserver access log. When test host / guest go to fetch things via HTTP OSSTest gets to know their status. I use Apache2 so I’ve set WebspaceLog to /var/log/apache2/access.log which just works.

Have Debian PXE installers ready. Remember the “DebianSuite” option in your config file? In order to make OSSTest fully functional you will also need to place Debian PXE installers in the right place. You can grab Debian’s PXE installers from any of the Debian archives around the world. And put them under the TFTP you just set up. I would recommend having at least amd64 and i386 in place. Make sure installers are accessible from the test host before proceeding.

By now we’re all set! Next step:


This will reset everything in standalone mode and create standalone.db, which includes test jobs and runtime variables. You can peek what’s inside that database with sqlite3.

The first job to run should be a build-* job. That can: 1) verify your setup is correct; 2) generate a bunch of runtime variables for subsequent test-* jobs.

./sg-run-job build-amd64 # WARNING: this will wipe out your test box

If the job pass, you’re all set. You can play around with other jobs. The default setting of jobs is to wipe out test box on every run. If you don’t want that you need to specify OSSTEST_HOST_REUSE=1 as stated in README.

An example of what I run:

./sg-run-job build-amd64-pvops
OSSTEST_HOST_REUSE=1 ./sg-run-job test-amd64-amd64-xl

If you only want to run a specific testcase, you can try OSSTEST_JOB=$JOBNAME ./ts-XXX host=$TESTHOSTNAME.

Customized tree / revisions

By default OSSTest always fetches trees and revisions from Xenbits. You can easily override them with standalone.config.

Say if I want to test a specific revision of Xen, I have:

export REVISION_XEN=c5e9596cd095e3b96a090002d9e6629a980904eb

in my standalone.confg.

You can look at make-flight to know all the interesting environment variables. (Sorry, no document yet)

Writing new testcases

If you’re interested in writing new testcase, you can do that in two simple steps:

  1. write ts-my-test-case script, you can use any existing testcase as template (they are prefixed with “ts-“)
  2. modify sg-run-job, which has the information for which testcases to run for a specific job

Do have a look as osstest/TestSupport.pm, in which you can find lots of helpers to accomplish your task.

Writing new test job

The script responsible for making jobs is cs-job-create. When you run OSSTest in standalone mode, it is probably more useful to modify make-flight. You also need to modify sg-run-job to link your new job with testcases.

Hope the above information can help you get started with OSSTest. If you have any problem, do come to Xen-devel and ask.

Some readers may recall the recent announcement of open-sourcing XenRT, and wondering about the relationship between OSSTest and XenRT. Long-term, we expect that XenRT will mature as an open development project and eventually displace OSSTest. But that is not likely to happen for another six months to a year. Developing OSSTest has benefits for the current development, and we hope that getting people involved in writing test cases for OSSTest will be of service in helping people write test cases for XenRT when it becomes available. So we have decided to continue to develop OSSTest until XenRT is ready to replace it.

Have fun! 🙂

QEMU vs. qemu-traditional update

Here is an update about feature completeness of QEMU compared to the old qemu-traditional.

But first, what is the difference between QEMU and qemu-traditional?

QEMU is the software that can be found at qemu.org, we can also call it QEMU upstream. It’s where all new features are supposed to land.

What we call “qemu-traditional” is the fork of QEMU that has been used by Xen. It became harder and harder to maintain and to upgrade with recent version QEMU, so we could not benefit from some of the new features that have been developed in QEMU, and also any bug that we would have found in the fork can be hard to fix upstream because the code would be very different.

So, we took everything that was needed from qemu-traditional to run QEMU with Xen and integrate them in a modern QEMU. Up to now, few features were missing to be able to use QEMU, but now, all the main features are in and QEMU became the default for most usage.

So what’s new in 4.3?

An important feature to be able to live-migrate a guest is a way to be able to track memory that has been modified during a live-migration. The feature first appears in Xen 4.2.2.

We also added the support to CPU hotplug for HVM as it was one of the missing features to get QEMU closer to qemu-traditional and have it as default.

Missing feature?

We are still missing a few things with QEMU upstream, so far, the limited support for VGA passthrough has not been upstream. Another missing feature would be the use of QEMU in a stub-domain instead of using qemu-traditional as it is right now. This last one is planned to be fixed in hopefully 4.4.

There is also patchs to fix the suspend resume cycle of an HVM guest that should be applied soon.

Beside those missing features, there are more works going one to enhance the support of many QEMU features that are not usable right now with the tool stack libXL, like using USB redirection or USB passthrough or even one day supporting QXL.

Evolving the Xen Project Test Infrastructure

One of the stated goals for 2013 and 2014 of the Xen Project Advisory Board is to Increase upstream Xen Hypervisor quality including the quality of its latest CPU and Platform features and to address problems with the code in a timely and proactive manner, including defects, security vulnerabilities and performance problems.

OSSTest and Coverity Scan

Earlier this year, first steps have been taken by making available the code for the OSSTest system (a smoke test system, which has been running since 2010). As a second step the Advisory Board worked with the Xen Project Technical Coordination Team, to add the Xen Hypervisor to the Coverity Scan static analysis service, that is open to open source projects with nonprofit status. There is still some detail to sort out on how we grant access to this resource: you can find the ongoing discussion here.

XenRT as basis for a comprehensive Test-as-a-Service Platform for the Xen Project

In the kick-off meeting for the Xen Project Advisory Board in April this year, Citrix offered to open source XenRT, the Citrix XenServer automated test system. This created the opportunity, to establish an independently hosted and expansive Test-as-a-Service Platform for the Xen Project.

Continue reading

SWIOTLB by Morpheus

The following monologue explains how Linux drivers are able to program a device when running in a Xen virtual machine on ARM.

The problem that needs to be solved is that Xen on ARM guests run with second stage translation in hardware enabled. That means that what the Linux kernel sees as a physical address doesn’t actually correspond to a machine address. An additional translation layer is set by the hypervisor to do the conversion in hardware.

Many devices use DMA to read or write buffers in main memory and they need to be programmed with the addresses of the buffers. In the absence of an IOMMU, DMA requests don’t go through the same physical to machine translation set by the hypervisor for virtual machines, devices need to be programmed with machine addresses rather than physical addresses. Hence the problem we are trying to solve.

Definitions of some of the technical terms used in this article are available at the bottom of the page.

Given the complexity of the topic, we decided to ask for help to somebody with hands-on experience with teaching the recognition of the differences between “virtual” and “real”.


At last.
Please. Come. Sit.
Xen Project Matrix

Do you realize that everything running on Xen is a virtual machine — that Dom0, the OS from which you control the rest of the system, is just the first virtual machine created by the hypervisor? Usually Xen assigns all the devices on the platform to Dom0, which runs the drivers for them.

I imagine, right now, you must be feeling a bit like Alice, tumbling down the rabbit hole?
Let me tell you why you are here.

You are here because you want to know how to program a device in Linux on Xen on ARM.

Continue reading