Author Archives: Ian Jackson

Security vs Features

We’ve just released a rather interesting batch of Xen security advisories. This has given rise in some quarters to grumbling around Xen not taking security seriously.

I have a longstanding interest in computer security. Nowadays I am a member of the Xen Project Security Team (the team behind security@xenproject, which drafts the advisories and coordinates the response). But I’m going to put forward my personal opinions.

Of course Invisible Things are completely right that security isn’t taken seriously enough. The general state of computer security in almost all systems is very poor. The reason for this is quite simple: we all put up with it. We, collectively, choose convenience and functionality: both when we decide which software to run for ourselves, and when we decide what contributions to make to the projects we care about. For almost all software there is much stronger pressure (from all sides) to add features, than to improve security.

That’s not to say that many of us working on Xen aren’t working to improve matters. The first part of improving anything is to know what the real situation is. Unlike almost all corporations, and even most Free Software projects, the Xen Project properly discloses, via an advisory, every vulnerability discovered in supported configurations. We also often publish advisories about vulnerabilities in other relevant projects, such as Linux and QEMU.

Security bugs are bugs, and over the last few years the Xen Project’s code review process has become a lot more rigorous. As a result, the quality of code being newly introduced into Xen has improved a lot.

For researchers developing new analysis techniques, Xen is a prime target. A significant proportion of the reports to security@xenproject are the result of applying new scanning techniques to our codebase. So our existing code is being audited, with a focus on the areas and techniques likely to discover the most troublesome bugs.

As I say, the Xen Project is very transparent about disclosing security issues; much more so than other projects. This difference in approach to disclosure makes it difficult to compare the security bug density of competing systems. When I worked for a security hardware vendor I was constantly under pressure to explain why we needed to do a formal advisory for our bugs. That is what security-conscious users expect, but our competitors’ salesfolk would point to our advisories and say that our products were full of bugs. Their products had no publicly disclosed security bugs, so they would tell naive customers that their products had no bugs.

I do think Xen probably has fewer critical security bugs than other hypervisors (whether Free or proprietary). It’s the best available platform for building high security systems. The Xen Project’s transparency is very good for Xen’s users. But that doesn’t mean Xen is good enough.

Ultimately, of course, a Free Software project like Xen is what the whole community makes it. In the project as a whole we get a lot more submissions of new functionality than we get submissions aimed at improving the security.

So personally, I very much welcome the contributions made by security-focused contributors – even if that includes criticism.

Xen Project automatic testing on community infrastructure

Currently the Xen Project’s automatic testing setup runs on a small set of hardware in space borrowed from Citrix. Because it’s on the Citrix network, it’s not possible to give access to other community members. The underlying systems are creaking rather. And the system is too small – we already find that testing is rather too slow.

The Xen Project Test Framework Working Group has agreed to press forward with a plan to provide a new setup (in a public colo, probably). We have a budget for this from the Advisory Board, which we think will be sufficient to provide a bigger and better setup than we have now.

We have decided to separate this immediately pressing concern – the inadequate and inaccessible hosting – from the longer-term questions of how to make more use of Xen community members’ existing test software. In particular, we have deferred the question of whether to stick with the existing osstest system long-term, or move to another system such as Citrix’s XenRT.

We’ll consider whether, when and how to make such a transition after we have sorted out our underlying infrastructure. We will make sure that the hardware and facilities we are organising now will be suitable for whatever software system we might want to run.

So, our immediate task now is to set out a more detailed plan for the amount and kind of hardware to acquire, and to identify a suitable hosting facility.

Debconf 13

I’ve recently returned from Debconf 13, in Vaumarcus in Switzerland. My colleague Ian Campbell joined me there.

Debconf is the annual conference for contributors to Debian, with a few hundred attendees. There’s a fairly standard conference format with a programme of talks and BoF sessions, but the best part of of a Debconf is usually the ad-hoc conversations with other developers. Often thorny design problems involving multiple parts of the system can be tackled much more effectively in person, so there’s quite a bit of vigorous handwaving and the odd whiteboard/flipchart session.

We had an excellent time and spent rather too much of it staring at the amazing view of Lake Neuchatel. Debian’s 20th birthday party was not to be missed either.

This year’s Debconf found a substantial offering of cloudy topics on the schedule. One major theme was the ways in which Debian are working on better integration with the big public clouds, for example by providing ready-to-use images and by better packing of cloud-related software.

Of particular interest for Xen was Thomas Goirand’s talk on the integration between OpenStack’s various components. OpenStack is a complicated piece of software which has been difficult to install and get running. Thomas, who runs a Xen-based public cloud provider, has been working to make the installation process smoother using Debian’s configuration management systems.

For me, an interesting topic was the continuing difficulty of integration between the Debian archive (Debian’s primary software repository) and git, and after a session in the bar with Joey Hess and others I wrote a tool to help with that.

Debconf is always a highlight of my year and I look forward to next year’s in Portland.

Xen is now officially in git

The Xen hypervisor is now officially in git.

There’s a single repo with a branch corresponding to each old xen*.hg tree:

  New git branch        Old mercurial tree
   master                xen-unstable.hg
   stable-4.0            xen-4.0-testing.hg
   stable-4.1            xen-4.1-testing.hg
   stable-4.2            xen-4.2-testing.hg
   staging               staging/xen-unstable.hg
   staging-4.0           staging/xen-4.0-testing.hg
   staging-4.1           staging/xen-4.1-testing.hg
   staging-4.2           staging/xen-4.2-testing.hg

Previously many (if not most) of the committers were using git themselves, and sometimes even inviting git-using contributors to provide their patch series as a public git branch.

While git’s user interface does have its critics (which include me!), its rich functionality, flexibility and performance are making it the power user’s dvcs of choice. Even if you accidentally shoot yourself in the foot (or vaporise your leg) due to some ill-considered command line syntax, there will be a way to recondense the vapour cloud back into a leg and stick it back on. As a friend of mine said recently, while you might either love or hate Marmite, with git many people both love and hate it.

So, anyway, after consultation and coordination, on Thursday we made the switch.

What used to be the official mercurial trees are now readonly mirrors. Contributors and downstreams who prefer to use mercurial can continue to use these, and of course submit patches using mercurial rather than git. So non-committers can indeed ignore this if they want to.

However, one benefit of officially switching to git is that there is now one very officially blessed git history. This means that it will be somewhat easier for git-using contributors to share patch series. But the main benefits are to committers.

The automatic testing system has needed some reeducation to cope, and is still suffering from some recidivism. The push gate for Xen 4.1 broke when it tried to push, and the bisector has been very confused. But this and any other problems should be sorted soon.

Personally, with my committer hat on, I’m already enjoying the convenience of having a single git tree containing all the Xen branches I deal with. And I’ve found that git’s tools for extracting patches from email and applying them are an improvement over what I was using before.
Continue reading

Xen automatic test system “osstest”

We do some automatic testing of the Xen hypervisor and tools branches. These tests form the “push gate” between the “staging” branch of xen-unstable (and the stable releases) and the non-staging branches.

Changes are committed to non-staging, and if the tests pass, the test system pushes them through to staging. So if a commit breaks one of the tests, the contributors don’t get to see their changes pushed and that tends of focus minds on fixing or reverting the problem commits. There’s also a cross-repository bisector which tries to automatically pin down which specific changeset is causing trouble.

I’ve been doing some work recently to disentangle this test system, which we now call “osstest”, from its supporting infrastructure. It’s now possible to use it standalone – for example, to reproduce failures which the test system has spotted, or to write new test cases.

Continue reading

xen-tools – a straightforward VM provisioning/installation tool

I’d like talk about the xen-tools package, which is found in Debian-derived distros. It’s a straightforward Xen VM provisioning tool with an unusual but attractive approach.

I use it in the automated testing system, for installing Debian-derived test VMs. And I run it by hand from the command line too.

What makes xen-tools special

What’s so different about xen-tools? Well, most VM provisioning tools arrange to run the guest’s copy of its own installer, in a fresh VM with a blank disk. They provide preseeding information with the answers to the questions that the installer asks. Another common approach is to have a blessed disk image, and make a guest by making a copy (perhaps a copy-on-write clone) of the master.

xen-tools doesn’t work like that. Instead, it relies on the existing Debian tools for installing chroots. chroots are a kind of lightweight near-virtualisation and are very heavily used by Debian’s developers to allow them to develop packages for different versions of the OS from the one they have installed (including perhaps different derivatives – so for example allowing packages for Ubuntu to be developed on a Debian machine or vice versa). Sometimes users find it chroots useful to gain access to different versions of software packages too.

xen-tools uses the chroot installation tool debootstrap: it sets up the disk area or LVM for the new VM, and then installs the new guest by running debootstrap in the management domain. The resulting approach is very simple compared to a VM-based run of the entire installer. There is no need to manage the booting of the installer, provide it with preseed information to configure it properly, and so forth. Logging and error handling are much improved. And you get pretty good control over the exact contents of the guest.

When should you choose xen-tools?

Firstly, xen-tools is aimed at systems administered from the command-line using xl/xm (perhaps with some management layer on top of that). xen-tools will write a domain configuration file suitable for use with xl or xm.

The biggest limitation is that it can only install a limited set of guests. At the time of writing the version of xen-tools in Debian testing can install most versions of Debian or Ubuntu, and also has support for CentOS 5 and 6. (The CentOS support is done using rinse rather than debootstrap.)

Continue reading