Category Archives: Security

Security vs Features

We’ve just released a rather interesting batch of Xen security advisories. This has given rise in some quarters to grumbling around Xen not taking security seriously.

I have a longstanding interest in computer security. Nowadays I am a member of the Xen Project Security Team (the team behind security@xenproject, which drafts the advisories and coordinates the response). But I’m going to put forward my personal opinions.

Of course Invisible Things are completely right that security isn’t taken seriously enough. The general state of computer security in almost all systems is very poor. The reason for this is quite simple: we all put up with it. We, collectively, choose convenience and functionality: both when we decide which software to run for ourselves, and when we decide what contributions to make to the projects we care about. For almost all software there is much stronger pressure (from all sides) to add features, than to improve security.

That’s not to say that many of us working on Xen aren’t working to improve matters. The first part of improving anything is to know what the real situation is. Unlike almost all corporations, and even most Free Software projects, the Xen Project properly discloses, via an advisory, every vulnerability discovered in supported configurations. We also often publish advisories about vulnerabilities in other relevant projects, such as Linux and QEMU.

Security bugs are bugs, and over the last few years the Xen Project’s code review process has become a lot more rigorous. As a result, the quality of code being newly introduced into Xen has improved a lot.

For researchers developing new analysis techniques, Xen is a prime target. A significant proportion of the reports to security@xenproject are the result of applying new scanning techniques to our codebase. So our existing code is being audited, with a focus on the areas and techniques likely to discover the most troublesome bugs.

As I say, the Xen Project is very transparent about disclosing security issues; much more so than other projects. This difference in approach to disclosure makes it difficult to compare the security bug density of competing systems. When I worked for a security hardware vendor I was constantly under pressure to explain why we needed to do a formal advisory for our bugs. That is what security-conscious users expect, but our competitors’ salesfolk would point to our advisories and say that our products were full of bugs. Their products had no publicly disclosed security bugs, so they would tell naive customers that their products had no bugs.

I do think Xen probably has fewer critical security bugs than other hypervisors (whether Free or proprietary). It’s the best available platform for building high security systems. The Xen Project’s transparency is very good for Xen’s users. But that doesn’t mean Xen is good enough.

Ultimately, of course, a Free Software project like Xen is what the whole community makes it. In the project as a whole we get a lot more submissions of new functionality than we get submissions aimed at improving the security.

So personally, I very much welcome the contributions made by security-focused contributors – even if that includes criticism.

The Bitdefender virtual machine introspection library is now on GitHub

The security threats we’re facing today are becoming increasingly sophisticated. Rootkits, and malware taking advantage of kernel and 0-day vulnerabilities pose especially serious challenges for classic anti-malware solutions, due to the latter’s lack of isolation: they’re typically executing in the same context as the malware they’re trying to prevent.


However, via Xen guest memory introspection, the anti-malware application can now live outside of the monitored guest, without relying on functionality that can be rendered unreliable by advanced malware (no agent or any other type of special software needs to run inside the guest). It does this by analyzing the raw memory of the guest OS, and identifying interesting kernel memory areas (driver objects and code, IDT, etc.) and user space memory areas (process code, stack, heap). This is made possible by harnessing existing hardware virtualization extensions (Intel EPT / AMD RVI). We hook the guest OS memory by marking interesting 4K pages as non-execute, or non-write, then audit accesses to those areas in code running in a separate domain.

Using hooks, we can detect that a driver (or kernel module) has been loaded or unloaded, that a new user process has been created, when user stack or heap is being allocated, and when memory is being paged in or out.


Using these techniques, we can protect against known rootkit hooking techniques, protect specific user processes against code injection, function detouring, code executing from the stack or heap areas, and unpacked malicious code. We can also inject remediation tools into the guest on-the-fly (with no help needed from within the guest).

In our quest for zero-footprint guest introspection using Xen, we needed to be able to decide whether changes to the guest OS are benign or malicious, and, once decided, to control whether they are allowed to happen, or rejected.

To that end, we needed a way to be notified of interesting Xen guest events, with a very low overhead, that we could use in dom0 (or a similarly privileged dedicated domain), and that would connect our introspection logic component to the guest.

We’re very happy to announce that the library we’ve created to help us perform virtual machine introspection, libbdvmi, is now on GitHub, under the LGPLv3 license, here. The library is x86-specific, and while there’s some #ifdeferry suggesting that the earliest supported version is Xen 4.3, only Xen 4.6 will work out-of-the-box with it. It has been written from scratch, using libxenctrl and libxenstore directly.

Xen 4.6 is an important milestone, as it has all the features needed to build a basic guest memory introspection application: in addition to support for emulating instructions in response to mem_access events and being able to suppress suspicious writes (which we’ve been able to contribute to 4.5), with the kind help of the Xen developer community we’ve now added support for XCR write vm_events, guest-requested custom vm_events, memory content hiding when instructed to emulate an instruction via a mem_access event response, and denying suspicious CR and MSR writes in response to a vm_event.

While LibVMI is great, and has been considered for the task, it has slightly different design goals:

  • in order to work properly, LibVMI requires that you install a configuration file into either $HOME/etc/libvmi.conf or /etc/libvmi.conf, with a set of entries for each virtual machine or memory image that LibVMI will access (while it is possible to use it without extracting this detailed information, doing so severely limits available guest information, and it does reflect a core design concern);
  • it uses Glib for caching (which gets indirectly linked with client applications, and so needs to be distributed with the application, or otherwise available on the machine where the application will be deployed);
  • it doesn’t publicly offer a way of mapping guest pages from userspace (so that we could write to them directly). For those technically inclined, something along the lines of XenDriver::mapVirtMemToHost();
  • it offers no way to ask for a page fault injection in the guest.

Libbdvmi aims to provide a very efficient way of working with Xen to access guest information in an OS-agnostic manner:

  • it only connects an introspection logic component to the guest, leaving on-the-fly OS detection and decision-making to it;
  • provides a xenstore-based way to know when a guest has been started or stopped;
  • has as few external library dependencies as possible – to that end, where LibVMI has used Glib for caching, we’ve only used STL containers, and the only other dependencies are libxenctrl and libxenstore;
  • allows mapping guest pages from userspace, with the caveat that this implies mapping a single page for writing, where LibVMI’s buffer-based writes would possibly touch several non-contiguous pages;
  • it works as fast as possible – we get a lot of events, so any unnecessary userspace / hypervisor context switches may incur unacceptable penalties (this is why one of our first patches had vm_events carry interesting register values instead of querying the quest from userspace after receiving the event);
  • last but not least, since the Xen vm_event code has been in quite a bit of flux, being able to immediately modify the library code to suit a new need, or to update the code for a new Xen version, has been a bonus to the project’s development pace.

We hope that the community will find it useful, and welcome discussion! It would be especially great if as much new functionality as possible makes its way into LibVMI.

Interested in learning more? Our Chief Linux Officer Mihai Dontu will be presenting “Zero-Footprint Memory Introspection with Xen” at LinuxCon North America on Wednesday, Aug. 19. Mihai is currently involved in furthering integrating Bitdefender hypervisor-based memory introspection technology with Xen. There are also two talks about Virtual Machine Introspection at the Xen Project Developer Summit: see Virtual Machine Introspection with Xen and Virtual Machine Introspection: Practical Applications.

Schrödinger’s Cat in a (Xen) Virtualzed ‘Box’

Fedora Logo

Fedora Logo

Yes, apparently Schrödinger’s cat is alive, as the latest release of Fedora – Fedora 19, codename Schrödinger’s cat– as been released on July 2nd, and that even happened quite on time.

So, apparently, putting the cat “in a box” and all the stuff was way too easy, and that’s why we are bringing the challenge to the next level: do you dare putting Schrödinger’s cat “in a virtual box”?

In other words, do you dare install Fedora 19 within a Xen virtual machine? And if yes, how about doing that using Fedora 19 itself as Dom0?
Continue reading Bugzilla Tracking

Henning Sprang, Mark Williamson, and I discussed the issue of people reporting bugs in the Bugzilla system with no guarantee that anyone was watching or working on the Bugzilla system. Several companies working on the Xen hypervisor are leveraging Bugzilla to track and monitor issues but there is no existing process to ensure that bugs entered by users or developers are being worked on. We are proposing the following process as a possible solution and will start this process next week:

· I will monitor the mailing list daily to see what bugs are entered.

· I will filter out all “corporate” bugs that are being entered by companies working on the hypervisor and will follow-up with the bug creator of non “corporate” bugs to ensure that the data entered is complete for a developer to understand and reproduce the problem.

· Mark will do a review of the bug reported to ensure that it is in fact a system issue which needs developer attention

· I will send an email 1x per week to mailing list with all the bugs that are open for developer attention; the complete information on the bugs per week will be in the Wiki at 

· Any developer interested in working on a bug from the list can either update Bugzilla themselves or work with me to ensure that the fix is documented and the bug is closed when work is complete

If you have any additional comments, please feel free to add them to this discussion on as I have sent this information to that mailing list or in the comments below.