Author Archives: George Dunlap

About George Dunlap

I received my PhD from the University of Michigan in December 2006. I joined XenSource in May of 2006. I've been working on analyzing performance in the Xen hypervisor. I'm currently working on a new scheduler.

Updates on XSA-213, XSA-214 and XSA-215

Today we released three patches for the following vulnerabilities: XSA-213, XSA-214 and XSA-215. Xen Project follows industry-accepted best practices regarding software security. This includes observing an embargo period, during which time the Xen Project security team will assess, respond, and prepare patches fixing the vulnerability, and distribute them privately to software and cloud providers before the public disclosure occurs.

When issuing a Xen Project Security Advisory (XSA), during the embargo this advisory is pre-disclosed to only members on the Xen Project Pre-Disclosure List. Vendors and open source projects who are on the Xen Project pre-disclosure list will not be affected by this security vulnerability and have updated their systems. The Xen Project security team has created fixes for these vulnerabilities, which can be publicly downloaded here: http://xenbits.xen.org/xsa/

Public cloud providers on the Xen Project predisclosure list were notified of these vulnerabilities two weeks ago; if your public cloud provider is on the list it is likely that your VMs are not vulnerable. Distributions and other software providers were also notified; they should have updated packages available soon (if they are not available already).

All three vulnerabilities have the potential to enable a guest virtual machine to break out of the hypervisor isolation. However, in order to exploit this vulnerability, an attacker would need to be running code in kernel mode of one or more VMs on the system. Any system that allows untrusted users to run arbitrary kernels will be particularly vulnerable.

Systems which only allow trusted users (such as IT professionals employed by the company) to run arbitrary kernels are less vulnerable, because an attacker would first need to find one or more exploit in the software running on one of the VMs before being able to then exploit this vulnerability. However, all users are encouraged to update as soon as possible.

Any 64-bit PV guest can exploit the vulnerability with XSA-213. The other two are more constrained. XSA-214 requires an attacker to control two different kinds of guests (either a PV one and an HVM one or a 32-bit PV one and 64-bit PV one). XSA-215 only affects you if your host has a very large amount of memory (either 3.5 TiB or 5 TiB depending on configuration).

Again, even with these constraints, we encourage you to update as soon as possible.

We take security very seriously and have developed security process best practices that are aimed for cloud environments that maximize fairness and transparency. We also have a very strict standard of review when it comes to new code being added to the Xen Project. We run Coverity static analyzer regularly to prevent certain classes of programming errors from being introduced. Additionally, we regularly run a generational fuzzing tool on our instruction emulator.

The Xen Project community developed Live Patching and introduced it into Xen Project 4.7. Now security fixes can be deployed without having to reboot VMs or have significant spare compute capacity to avoid reboots via VM migration.

These vulnerabilities were discovered by Jann Horn, from Google Project Zero.

Request for Comment: Scope of Vulnerabilities for which XSAs are issued

Issuing advisories has a cost: It costs the security team significant amounts of time to craft and send the advisories; it costs many of our downstreams time to apply, build, and test patches; and it costs many of our users time to decide whether to do an update, and if so, to test and deploy it.

Given this, the Xen Project Security Team wants to clarify when they should issue an advisory or not: the Xen Security Response Process only mentions “‘vulnerabilities”, without specifying what constitutes a vulnerability.

We would like guidelines from the community about what sorts of issues should be considered security issues (and thus will have advisories issued). I have posted the second version a draft of a section I am proposing to be added to the Xen Security Policy to xen-devel; a copy is included below for your convenience. There are only minor modifications from the first draft, so barring major feedback from the wider community it will likely achieve consensus. If you want input, now is the time to speak up.

Most of it is just encoding long-established practice. But there are two key changes and / or clarifications that deserve attention and discussion:

    Criteria 2c: Leaking of mundane information from Xen or dom0 will not be considered a security issue unless it may contain sensitive guest or user data

Criteria 4: If no operating systems are vulnerable to a bug, no advisory will be issued.

If you want to weigh in on the question, please join the discussion on xen-devel before 28 February. The title of the thread is “RFC v2: Scope of Vulnerabilities for which XSAs are issued”.

Continue reading

XSA-108: Not the vulnerability you’re looking for

There has an unusual amount of media attention to XSA-108 during the embargo period (which ended Wednesday) — far more than any of the previous security issues the Xen Project has reported. It began when a blogger complained that Amazon was telling customers it would be rebooting VMs in certain regions before a specific date. Other media outlets picked it up and noticed that the date happened to coincide with the release of XSA-108, and conjectured that the reboots had something to do with that. Soon others were conjecturing that, because of the major impact to customers of rebooting, that it must be something very big and important, similar to the recent Heartbleed and Shell Shock vulnerabilities. Amazon confirmed that the reboots had to do with XSA-108, but could say nothing else because of the security embargo.

Unfortunately, because of the nature of embargoes, nobody with any actual knowledge of the vulnerability was allowed to say anything about it, and so the media was entirely free to speculate without any additional information to ground the discussion in reality.

Now that the embargo has lifted, we can talk in detail about the vulnerability; and I’m afraid that people looking for another Shell Shock or Heartbleed are going to be disappointed. No catchy name for this one.

Continue reading

The Docker exploit and the security of containers

We normally only cover news and information directly related to Xen in this channel, but we thought it might be useful to briefly expand our scope a bit to mention the recent discussion about the Docker security exploit.

What’s the news?

Well to begin with, a few weeks ago Docker 1.0 was released, just in time for DockerCon.

Then last week, with timing that seems rather spiteful, someone released an exploit that allows a process running as root within a Docker container to break out of a Docker container.

I say it was a bit spiteful, because as the post-mortem on Docker’s site demonstrates, it exploits a vulnerability which was only present until Docker 0.11; it was fixed in Docker 0.12, which eventually became Docker 1.0.

Nonetheless, this kicked off a bit of a discussion about Docker, Linux containers, and security in several places, including Hacker News, the Register, seclists.org, and the OSv blog.

There is a lot of interesting discussion there. I recommend skimming through the Hacker News discussion in particular, if you have the time. But I think the comment on the Hacker News discussion by Solomon Hykes from Docker, puts it best:

As others already indicated this doesn’t work on 1.0. But it could have. Please remember that at this time, we don’t claim Docker out-of-the-box is suitable for containing untrusted programs with root privileges. So if you’re thinking “pfew, good thing we upgraded to 1.0 or we were toast”, you need to change your underlying configuration now. Add apparmor or selinux containment, map trust groups to separate machines, or ideally don’t grant root access to the application.

I’ve played around with Docker a little bit, and it seems like an excellent tool for packaging and deploying applications. Using containers to separate an application from the rest of the user-space of your distribution, exposing it only to the very forward-compatible Linux API is a really clever idea.

However, using containers for security isolation is not a good idea. In a blog last August, one of Docker’s engineers expressed optimism that containers would eventually catch up to virtual machines from a security standpoint. But in a presentation given in January, the same engineer said that the only way to have real isolation with Docker was to either run one Docker per host, or one Docker per VM. (Or, as Solomon Hykes says here, to use Dockers that trust each other in the same host or the same VM.)

We would concur with that assessment.

Release management: Risk, intuition, and freeze exceptions

I’ve been release coordinator for Xen’s 4.3 and 4.4 releases. For the 4.5 release, I’ve handed this role off to Konrad Wilk, from Oracle. In this blog, I try to capture some of my thoughts and experience about one aspect of release management: deciding what patches to accept during a freeze.

I have three goals when doing release management:

  1. A bug-free release
  2. An awesome release
  3. An on-time release

One of the most time-consuming seasons of being a release manager is during any kind of freeze. You can read in detail about our release process elsewhere; I’ll just summarize it here. During normal development, any patch which has the approval of the relevant maintainer can be accepted. As the release approaches, however, we want to start being more and more conservative in what patches we accept.

Obviously, no patch would ever be considered for acceptance which didn’t improve Xen in some way, making it more awesome. However, it’s a fact of software that any change, no matter how simple or obvious, may introduce a bug. If this bug is discovered before the release, it may delay the release, making it not on-time; or it may not be found until after the release, making the release not bug-free. The job of helping decide whether to take a patch or not falls to the release coordinator.

But how do you actually make decisions? There are two general things to say about this.

Risk

The only simple rule to follow that will make sure that there are no new bugs introduced is to do no development at all. Since we must do development, we have to learn to deal with risk.

Making decisions about accepting or rejecting patches as release coordinator is about making calculated risks: look at the benefits, look at the potential costs, look at the probabilities, and try to balance them the best you can.

Part of making calculated risks is accepting that sometimes your gamble won’t pay off. You may approve a patch to go in, and it will then turn out to have a bug in it which delays the release. This is not necessarily a failure: if you can look back at your decision and say with honesty, “That was the best decision I could have made given what I knew at the time”, then your choice was the right one.

The extreme example of this kind of thinking that of a poker player: a poker player may make a bet that she knows she only has a 1 in 4 chance of winning, if the pay-off is more than 4 to 1; say, 5 to 1. Even though she loses 75% of the time, the 25% that she does win will pay for the losses. And when she makes the bet and loses (as she will 75% of the time), she knows she didn’t make a mistake; taking risks is just a part of the game.

Obviously as release coordinator, the costs of a bug are generally higher than the benefits of a feature. But the general principle — of taking calculated risks, and accepting occasional failure as the inevitable consequence of doing so — applies.

Intuition

But how do we actually calculate the risks? A poker player frequently deals with known quantities: she can calculate that there is exactly a 25% chance of winning, exactly a 5x monetary pay-off, and do the math; the release coordinator’s decisions are not so quantifiable.

shutterstock_58404295This is where intuition comes in. While there are a handful of metrics that can be applied to patches (e.g., the number of lines in the patch), for the most part the risk and benefit are not very quantifiable at all: expert judgement / intuition is the only thing we have.

Now, research has shown that the intuition of experts can, under the right circumstances, be very good. Intuition can quickly analyze hundreds of independent factors, and compare against thousands of instances, and give you a result which is often very close to the mark.

However, research has also shown that in other circumstances, expert intuition is worse than random guessing, and far worse than a simple algorithm. (For a reference, see the books listed at the bottom.)

One of the biggest ways intuition goes wrong is by only looking at part of the picture. It is very natural for programmers, when looking at a patch, to consider only the benefits. The programmer’s intuition then accurately gives them a good sense of the advantage of taking the patch; but doesn’t warn them about the risk because they haven’t thought about it. Since they have a positive feeling, then they may end up taking a patch even when it’s actually too risky.

The key then is to make sure that your intuition considers the risks properly, as well as the benefits. To help myself with this, during the 4.4 code freeze I developed a sort of checklist of things to think about and consider. They are as follows:

  1. What is the benefit of this patch?
  2. What is the probability this patch has a bug?
  3. If this patch had a bug, what kind of bug might it be?
  4. If this patch had a bug, what is the probability we would find it
    before the release?

When considering the probability of a bug, I look at two things:

  • The complexity of the patch
  • My confidence in my / the other reviewers’ judgement.

Sometimes you’re looking at code you’re very familiar with or is straightforward; sometimes you’re looking at code that is very complicated or you’re not that familiar with. If the patch looks good and it’s code you’re familiar with, it’s probably fine. If the patch looks good but it’s code you’re not familiar with, there’s a risk that your judgement may be off.

When trying to think of what kind of bug it might be, I look at the code that it’s modifying, and consider things on a spectrum:

  1. Error / tool crash on unexpected trusted input; or normal input to library-only commands
  2. Error / tool crash on normal input, secondary commands / new functionality
  3. Error / tool crash on normal input, core commands
  4. Performance
  5. Guest crash / hang
  6. Host crash / hang
  7. Security vulnerability
  8. Data loss

Usually you can tell right away where in the list a bug might be. Modifying xenpm or xenctx? 3 max. Modifying the scheduler? Probably #6. Modifying hypercalls available to guests? #7. And so on.

When asking whether we’d find the bug before the release, consider the kind of testing the codepath is likely to get. Is it tested in osstest? In XenRT? Or is it in a corner case that few people really use?

After thinking through those four questions, and going over the criteria in detail, then my intuition is probably about as well-formed as it’s going to get. Now I ask the fifth question: given the risks, is it worth it to accept this patch?

After giving it it some thought, I went with my best guess. Sometimes I’m just not sure; in which case go away and do something else for a couple of hours, then come back to it (going over again the four questions to make sure they’re fresh in my mind). The first few dozen times this took a very long time; as I gained experience, judgements came faster (although many were still painfully slow).

In some cases, I just didn’t have enough knowledge of the code to make the judgement myself; this happened once or twice with the ARM code in the 4.4 release. In that case, my goal was to try to make sure that those who did have the relevant knowledge were making sound decisions: thinking about both the benefits and the risks and weighing them appropriately.

For those who want to look further into risk and intuition, several books have had a pretty big influence on my thinking in this area. Probably the best one, but also the hardest (most dense) one, is Thinking, Fast and Slow, by Daniel Kahneman. It’s very-well written and accessible, but just contains a huge amount of information that is different to the way you normally think. Not a light read. Another one I would recommend is The Black Swan, by Nassim Nicholas Taleb. And finally, Blink, by Malcolm Gladwell.

Xen 4.4 Released

Xenproject.org is pleased to announce the release of Xen 4.4.0. The release is available from the download page:

Xen 4.4 is the work of 8 months of development, with 1193 changesets. It’s our first release made with an attempt at a 6-month development cycle. Between Christmas, and a few important blockers, we missed that by about 6 weeks; but still not too bad overall.

Additionally, this cycle we’ve had a massive increase in the amount of testing. The XenProject’s regression testing system, osstest, has recieved a number of additional tests, and the XenServer team at Citrix have put Xen through their massive testing suite (XenRT). Additionally, early in this development cycle we had the go-ahead to use Coverity static analysis engine to comb through the source code for hard-to-spot bugs. The result should be that Xen 4.4 is one of the most secure, reliable releases yet.

Highlights

Although the development part of the release cycle was shorter than the previous one, we still have far too many exciting improvements than we can mention in this blog post; I’ll call out just a few.

Probably one of the most important is solid libvirt support for libxl. Jim Fehlig from SuSE and Ian Jackson from Citrix worked together to test and improve the interface between libvirt and libxl, making it fast and reliable. This lays the foundation for solid integration into any tools that can use libvirt, from GUI VM managers to cloud orchestration layers like CloudStack or OpenStack.

Another big one is a new scalable event channel interface, designed and implemented by David Vrabel from Citrix. The original Xen event channel interface was limited to the number of bits on the platform squared — 1024 for 32-bit guests and 4096 for 64-bit guests. With many VMs requiring 4 event channels each, that means a theoretical maximum of 256 guests on a 32-bit dom0 — more than enough back when a large machine had 8 cores, and every VM was a full OS; but a major limitation on systems with 128 cores, or those using cloud OSes like Mirage or OSv. The new “FIFO” event channel interface by default scales up to over 200,000 event channels, and in the future can be extended even further if necessary in a backwards-compatible manner. This should be enough for many years to come.

The ARM port is maturing quickly. As of 4.4, the hypervisor ABI for ARM has been declared stable, meaning that any guest which uses the 4.4 ARM ABI can rely on being able to boot on all future versions of Xen. There are a number of improvements making Xen on ARM more flexible, easier to set up and use, and easier to extend to new platforms. More details can be found in the Xen 4.4 feature list.

One other feature worth a note is Nested Virtualization on Intel hardware. It’s not ready for production use yet, but it has improved to the point where we feel comfortable moving it from “experimental” to “tech preview”. Please feel free to take it for a spin and report any issues you find.

There are many more improvements and changes under the hood. For a more complete list, see the Xen 4.4 feature list.

Features in related projects

The Xen Project is part of a much larger ecosystem of projects. We are typically very closely tied to Linux and qemu, but a number of other projects have had important developments that are worth a mention.

The first is the pv port of grub2. Rather than having a re-implementation of grub in the Xen tree, grub2 now has native support for running in Xen and using the Xen pv block protocol. This guarantees 100% compatibility with grub2 going forward.

Another project worth a mention is the 3.3 release of Xen Orchestra. Xen Orchestra is a web interface that interfacer with the xapi protocol (and thus can be used for XCP, XenServer, or other xapi-based systems). New creating snapshots, revert or delete, remove host from pool, restart toolstack/reboot/shutdown host) and more stable upgrade process from appliance

Finally, GlusterFS 3.5 now supports creating iSCSI nodes. One of the benefits of this is that now, by creating iSCSI devices in dom0, Xen guest disks can be stored in GlusterFS.

Updates