Author Archives: zkeaton

Xen Project Hypervisor Power Management: Suspend-to-RAM on Arm Architectures

This is the second part of the Xen Project Hypervisor series on power management. The first article focused on how virtualization and power management are coalescing into an energy-aware hypervisor.

In this post, the focus is on a project that was started to lay the foundation for full-scale power management for applications involving the Xen Project Hypervisor on Arm architectures. The group intends to make Xen on Arm’s power management the open source reference design for other Arm hypervisors in need of power management capabilities. Read the full story via here.

Xen Project Matrix

Xen Project Hypervisor: Virtualization and Power Management are Coalescing into an Energy-Aware Hypervisor

Power management in the Xen Project Hypervisor historically targets server applications to improve power consumption and heat management in data centers reducing electricity and cooling costs. In the embedded space, the Xen Project Hypervisor faces very different applications, architectures and power-related requirements, which focus on battery life, heat, and size.

Although the same fundamental principles of power management apply, the power management infrastructure in the Xen Project Hypervisor requires new interfaces, methods, and policies tailored to embedded architectures and applications. This post recaps Xen Project power management, how the requirements change in the embedded space, and how this change may unite the hypervisor and power manager functions. Read the full article on here.

Improving the Stealthiness of Virtual Machine Introspection on Xen

This blog post comes from Stewart Sentanoe of the University of Passau. Stewart is a PhD student and he was recently a Google Summer of Code Intern working on the Honeynet Project. 

Project Introduction

Virtual Machine Introspection

Virtual Machine Introspection (VMI) is the process of examining and monitoring a virtual machine from the hypervisor or virtual machine monitor (VMM) point of view. Using this approach, we can get the untainted information of the monitored virtual machine. There are three main problems with VMI currently:

  • Semantic gap: How do you interpret low level data into useful information?
  • Performance impact: How big is the overhead?
  • Stealthiness: How to make the monitoring mechanism hard to be detected by the adversary?

This project focused on the third problem, and specifically on how to hide the breakpoint that has been set. We do not want the adversary to be able to detect whether there is a breakpoint that has been set to some of the memory addresses. If they are able to detect the breakpoint, most likely the adversary will not continue the attack and we will learn nothing. By leveraging VMI, we are able to build high interaction honeypot where the adversary can do whatever they want with the system. Thus, we can gather as much information as we can and we get the big picture of what’s going on in the system and learn from it.

Setting a Breakpoint Implemented by Drakvuf

DRAKVUF is a virtualization based agentless black-box binary analysis system developed by Tamas K Lengyel. DRAKVUF allows for in-depth execution tracing of arbitrary binaries (including operating systems), all without having to install any special software within the virtual machine used for analysis ( and

There are two ways to set a breakpoint implemented by DRAKVUF using INT3 (0xCC opcode) and Xen altp2m.

These are the following steps by DRAKVUF to inject breakpoint using INT3:

  1. Inject 0xCC into the target
  2. Mark pages Execute-only in the EPT (Extended Page Tables)
  3. If anything tries to read the page:
    1. Remove 0xCC and mark page R/W/X
    2. Singlestep
    3. Place 0xCC back and mark page X-only
  4. When 0xCC traps to Xen
    1. Remove 0xCC
    2. Singlestep
    3. Place 0xCC back

Sounds good right? But, there is a big problem when INT3 is used.

To make the breakpoint mechanism work well with multiple vCPUs, DRAKVUF uses Xen altp2m. At the normal runtime of a VM, each guest’s physical memory (GFN – Guest Frame Number) will be mapped one to one to the machine (host) physical memory (MFN – Machine Frame Number) as shown in the image below.

Next, to set a breakpoint, DRAKVUF will copy the target page to the end of the guest’s physical memory and add the trap there. DRAKVUF will also make an empty page (the purposes will be explained later) as shown below.

Now, during the runtime, the pointer of the original target will be switched h to the copy page as shown below and marked as execute only.

If a process tries to execute those pages, it can simply switch the pointer back to the original, single step and then switch the pointer to the copy page again. You might be thinking that if an adversary is able to scan “beyond” the physical memory, the adversary will detect a page that contains the copy. This where the empty page kicks in, whenever a process tries to read or write to the copy page, DRAKVUF will simply change the pointer to the empty page as shown below.

Sounds cool doesn’t it? Of course it is! But, there are several problems with this process, which led to this GSOC project. The sections below will cover them piece by piece.  

Problems of DRAKVUF

There are three problems that I found out during this project:

  1. There’s a M:1 relation between the shadow copy and the empty page, which means that if we set breakpoints to two addresses, it will create two shadow copy and only one empty page.
  2. If an adversary is able to write “something” to a shadow copy, the “something” will also appear on the other shadow copy which can raise their suspicious level.
  3. The current implementation of DRAKVUF will use ’00’  for the empty page, but the real behaviour never been observed.

Proposed Milestones

There are two milestones for this project:

  1. Create a proof of concept (kernel module) that detects the presence of DRAKVUF by trying to write “something” to one of the shadow copy and probe the second shadow copy to check the existence of the “something”
  2. Patch DRAKVUF

The Process and the Findings

At the beginning of this project, I had no idea how to read the memory beyond the physical address space, but then I found this article which describes a function (ioremap) that I used for my kernel module (available here). The drawback is that it requires some debug information generated by DRAKVUF, for example the address of the shadow copy.

When I executed the code without the writing part, I got this:

As expected, it gave me empty result. Then, when I wrote “something” to the first address which in this point is letter ‘A’ (in hex is 41). The ‘A’ also appears on the second address:

Bingo! Something fishy there. Alright, then I tried to print more addresses:

Did you see something weird there? Yes, the ‘FF’, actually the empty is ‘FF’ instead ’00’. So actually, an adversary does not need to write “something” to the empty page, it just simply detects if there are ’00’ then it reveals the presence of DRAKVUF.

But where is the ‘FF’ comes from? Architecturally, all physical addresses defined by CPUID EAX=80000008h bits 15-8 (more here) are considered “valid” In Linux, it checks the address validity when it sets up the memory page table (see here). It is up to the firmware to tell the OS and the hypervisor what range are valid with the E820 map (see here). When a process requests a memory address that is not valid (assuming the new page table is made), it goes through the Memory Management Unit (MMU) and then Platform Controller Hub (PCH). The PCH tries to find the valid physical memory but could not found it then, if it involves write, the written value will be ignored and if it involves read, it will return all 1s. This behaviour is written into this (page 32) Intel document and anyway VMI (for now) just works on Intel processor.

Alright, now time to fix this.

First is pretty easy where I just write ‘FF’ to the shadow page:

It solved the simple problem. But now let’s talk about the bigger problem about the writing. The idea is to simply ignore write attempt to the shadow page and also to the empty page. For both cases, we can use features provided by Xen, which emulate the write. Sounds easy, but actually there was another problem: LibVMI (library that used by DRAKVUF) does not support the write emulation flag, so I needed to patch it up (see here).

Alright, now I check whenever a process tries to write to the shadow copy, then just do the emulation:

Now, we also need to add TRAP to the shadow copy so we can also do emulation whenever a process tries to write to it.

Now every time a process tries to write to either the empty page and the shadow copy, the written value will be not “stored” in the memory. Thus, it hides DRAKVUF better.


This project increases the stealthiness level of DRAKVUF. With a high level of stealthiness, it opens up the potential for a new generation honeypots, intrusion detection systems and dynamic malware analysis where it will be hard for the adversary to detect the presence of the monitoring system.


Thanks to Tamas K Lengyel and Varga-Perke Balint you rock! Thank you for your help and patience. Thank you also for Benjamin Taubmann for the support and of course Honeynet and Google for GSOC 2018 🙂



Xen Project Announces Schedule for its Annual Developer and Design Summit

Today, we are excited to announce the program and speakers for the Xen Project Developer and Design Summit. The summit brings together developers, engineers, and Xen Project power users for in-person collaboration and educational presentations. The event will take place in Nanjing Jiangning, China from June 20 -22, 2018.

This is the fifth annual Xen Project Summit with presentations and panels focusing on hypervisor performance and development, security, automotive and much more. The conference will kick-off with a weather report from Lars Kurth, chairperson of the Xen Project and director of open source at Citrix.

At last year’s Xen Project Developer Summit in Budapest, Hungary.

A sample of presentations include:

  • Sung-Min Lee, principal engineer at Samsung Electronics, will present a production-ready automotive virtualization solution with Xen.
  • Marek Marczykowski-Górecki, senior systems developer, Invisible Things Lab, will present on linux-based device model stubdomains in Qubes OS.
  • Julien Grall, senior software virtualization engineer at Arm, will share capabilities that were added to the latest revision of the ARmv7-A architectures and how Arm has been improving virtualization support with incremental versions of the Armv8 architecture.
  • Felipe Huici, chief researcher at NEC, and Florian Schmidt, research scientist at NEC, will co-present on Unikraft, a sub-project of the Xen Project aimed at automativing the process of building customized unikernels tailored to a specific applications.
  • Bo Zhang, business analyst at Huawei, will introduce Huawei Cloud’s optimization on the Xen platform to solve regular problems that occur in customer scenarios

You can view the full schedule here.

Beyond panels and presentations, the Xen Project will be running design sessions that share a similar format to Xen Project hackathons. Attendees of the conference have the opportunity to propose design sessions during the conference. Current design topics already include Making Safety Certifications for Xen Easier, From Hobbyist to Maintainer: Why and How and Reworking x86 in Xen (Current and Future Plans).

If you’ve never attended a Xen Project Developer and Design Summit, check out last year’s presentations to get a better feel for the event.

A special thank you Citrix for being a diamond sponsor of the summit.


Summer = Xen Project Internships!

We received a lot of amazing submissions for our summer Outreachy internship program and have accepted Dafna Hirschfeld to join us in creating new execution targets for Unikraft. Unikraft is a Xen Project incubation project that aims to simplify the process of building unikernels through a unified and customizable code base.

Currently, Unikraft supports building images that can be executed as a virtual machine on Xen and KVM, and as an ELF binary within the Linux user space environment. Support for more execution targets is done by providing more platform libraries that can be chosen during build. Dafna will be fairly free to choose a platform, perhaps based on familiarity or curiosity. Examples for platform choice include bare-metal, ARM, X86_64, VMware, Microsoft Hyper-V, and bhyve.

If you are unfamiliar with Outreachy, it provides three-month internships for people from groups traditionally underrepresented in tech. Interns work remotely with mentors from Free and Open Source Software (FOSS) communities. Xen Project interns have later gone on to work at companies like Oracle, Google and Citrix.

In addition to Outreachy, we are excited to announce a few new interns that will be working on the Xen Project hypervisor through Google Summer of Code. Although the Xen Project was not a mentoring organisation for Google Summer of Code this year, FreeBSD and The Honeypot Project were and had a number of Xen Project related projects. Google Summer of Code is a global program focused on bringing more student developers into open source software development.

Interns that are a part of the Google Summer of Code and working on pushing Xen Project technologies forward include:

  • Kristaps Civkulis who will be working on enabling the EFI loader to load FreeBSD Xen Dom0. There are two parts to the project – you can learn more about it here. The organization supporting this is the FreeBSD Project.
  • Pratyush Yadav who will import the Xen grant-table bus_dma(9) handlers from OpenBSD. FreeBSD Project is supporting Pratyush.
  • Lele Ma who will Port LibVMI to Xen MiniOS. In this project, the core functionalities of the LibVMI will be ported to Xen MiniOS. After ported, Xen MiniOS will have the basic capabilities of introspecting the memory of other guest virtual machines. Honeynet Project is supporting Lele.
  • Honeynet Project is also supporting Stewart Sentanoe who is working on stealth monitoring with Xen altp2m based on previous work that has been done – see here. And Ulrich Fourier who is working on adding support for ARM introspection, which is a follow-up to a 2016 GSoC project that developed altp2m support to Xen on ARM.

Working in open source is a great way to start your career in technology. In a recent survey from HackerRank 84% of respondents (including CEOs, CTOs and company founders) said they look to an applicant’s GitHub project work as an indicator of a prospective employee’s on-the-job skills.

We want to thank everyone who applied to our Outreachy scholarship, and look forward to sharing the accomplishments of our interns. Welcome to open source!


Join us at Root Linux Conference Happening in Kyiv, Ukraine This April!

Root Linux Conference is coming to Kyiv, Ukraine on April 14th. The conference is the biggest Linux and embedded conference in Eastern Europe with presenters exploring topics like: Linux in mobile devices, wearables, medical equipment, vehicles, and more. Want to learn about the next generation of embedded solutions? This is the conference for you.

Juergen Gross, Linux Kernel developer at SUSE, and Paul Durrant, Senior Principal Software Engineer at Citrix Systems, are keynoting the conference. Juergen will cover Xen paravirtualized (PV) devices and Paul will cover Intel GVT-g integration into XenServer.

Early bird priced tickets for the event are still available here.

See you there!