Tag Archives: Internships

Improving the Stealthiness of Virtual Machine Introspection on Xen

This blog post comes from Stewart Sentanoe of the University of Passau. Stewart is a PhD student and he was recently a Google Summer of Code Intern working on the Honeynet Project. 

Project Introduction

Virtual Machine Introspection

Virtual Machine Introspection (VMI) is the process of examining and monitoring a virtual machine from the hypervisor or virtual machine monitor (VMM) point of view. Using this approach, we can get the untainted information of the monitored virtual machine. There are three main problems with VMI currently:

  • Semantic gap: How do you interpret low level data into useful information?
  • Performance impact: How big is the overhead?
  • Stealthiness: How to make the monitoring mechanism hard to be detected by the adversary?

This project focused on the third problem, and specifically on how to hide the breakpoint that has been set. We do not want the adversary to be able to detect whether there is a breakpoint that has been set to some of the memory addresses. If they are able to detect the breakpoint, most likely the adversary will not continue the attack and we will learn nothing. By leveraging VMI, we are able to build high interaction honeypot where the adversary can do whatever they want with the system. Thus, we can gather as much information as we can and we get the big picture of what’s going on in the system and learn from it.

Setting a Breakpoint Implemented by Drakvuf

DRAKVUF is a virtualization based agentless black-box binary analysis system developed by Tamas K Lengyel. DRAKVUF allows for in-depth execution tracing of arbitrary binaries (including operating systems), all without having to install any special software within the virtual machine used for analysis (https://drakvuf.com and https://github.com/tklengyel/drakvuf).

There are two ways to set a breakpoint implemented by DRAKVUF using INT3 (0xCC opcode) and Xen altp2m.

These are the following steps by DRAKVUF to inject breakpoint using INT3:

  1. Inject 0xCC into the target
  2. Mark pages Execute-only in the EPT (Extended Page Tables)
  3. If anything tries to read the page:
    1. Remove 0xCC and mark page R/W/X
    2. Singlestep
    3. Place 0xCC back and mark page X-only
  4. When 0xCC traps to Xen
    1. Remove 0xCC
    2. Singlestep
    3. Place 0xCC back

Sounds good right? But, there is a big problem when INT3 is used.

To make the breakpoint mechanism work well with multiple vCPUs, DRAKVUF uses Xen altp2m. At the normal runtime of a VM, each guest’s physical memory (GFN – Guest Frame Number) will be mapped one to one to the machine (host) physical memory (MFN – Machine Frame Number) as shown in the image below.

Next, to set a breakpoint, DRAKVUF will copy the target page to the end of the guest’s physical memory and add the trap there. DRAKVUF will also make an empty page (the purposes will be explained later) as shown below.

Now, during the runtime, the pointer of the original target will be switched h to the copy page as shown below and marked as execute only.

If a process tries to execute those pages, it can simply switch the pointer back to the original, single step and then switch the pointer to the copy page again. You might be thinking that if an adversary is able to scan “beyond” the physical memory, the adversary will detect a page that contains the copy. This where the empty page kicks in, whenever a process tries to read or write to the copy page, DRAKVUF will simply change the pointer to the empty page as shown below.

Sounds cool doesn’t it? Of course it is! But, there are several problems with this process, which led to this GSOC project. The sections below will cover them piece by piece.  

Problems of DRAKVUF

There are three problems that I found out during this project:

  1. There’s a M:1 relation between the shadow copy and the empty page, which means that if we set breakpoints to two addresses, it will create two shadow copy and only one empty page.
  2. If an adversary is able to write “something” to a shadow copy, the “something” will also appear on the other shadow copy which can raise their suspicious level.
  3. The current implementation of DRAKVUF will use ’00’  for the empty page, but the real behaviour never been observed.

Proposed Milestones

There are two milestones for this project:

  1. Create a proof of concept (kernel module) that detects the presence of DRAKVUF by trying to write “something” to one of the shadow copy and probe the second shadow copy to check the existence of the “something”
  2. Patch DRAKVUF

The Process and the Findings

At the beginning of this project, I had no idea how to read the memory beyond the physical address space, but then I found this article which describes a function (ioremap) that I used for my kernel module (available here). The drawback is that it requires some debug information generated by DRAKVUF, for example the address of the shadow copy.
https://gist.github.com/scorpionzezz/d4f19fb9cc61ff4b50439e9b1846a17a

When I executed the code without the writing part, I got this: https://gist.github.com/scorpionzezz/6e4bdd0b22d5877057823a045c784721

As expected, it gave me empty result. Then, when I wrote “something” to the first address which in this point is letter ‘A’ (in hex is 41). The ‘A’ also appears on the second address: https://gist.github.com/scorpionzezz/ce6623f1176e99de61617222ceba462a

Bingo! Something fishy there. Alright, then I tried to print more addresses: https://gist.github.com/scorpionzezz/22bdb3c727dd130bb59b28cf717d9bac

Did you see something weird there? Yes, the ‘FF’, actually the empty is ‘FF’ instead ’00’. So actually, an adversary does not need to write “something” to the empty page, it just simply detects if there are ’00’ then it reveals the presence of DRAKVUF.

But where is the ‘FF’ comes from? Architecturally, all physical addresses defined by CPUID EAX=80000008h bits 15-8 (more here) are considered “valid” In Linux, it checks the address validity when it sets up the memory page table (see here). It is up to the firmware to tell the OS and the hypervisor what range are valid with the E820 map (see here). When a process requests a memory address that is not valid (assuming the new page table is made), it goes through the Memory Management Unit (MMU) and then Platform Controller Hub (PCH). The PCH tries to find the valid physical memory but could not found it then, if it involves write, the written value will be ignored and if it involves read, it will return all 1s. This behaviour is written into this (page 32) Intel document and anyway VMI (for now) just works on Intel processor.

Alright, now time to fix this.

First is pretty easy where I just write ‘FF’ to the shadow page: https://gist.github.com/scorpionzezz/9853d836b38b82c2961c1d437390c8a3

It solved the simple problem. But now let’s talk about the bigger problem about the writing. The idea is to simply ignore write attempt to the shadow page and also to the empty page. For both cases, we can use features provided by Xen, which emulate the write. Sounds easy, but actually there was another problem: LibVMI (library that used by DRAKVUF) does not support the write emulation flag, so I needed to patch it up (see here).

Alright, now I check whenever a process tries to write to the shadow copy, then just do the emulation: https://gist.github.com/scorpionzezz/3a12bebdd43d5717d671136f0fc0069c

Now, we also need to add TRAP to the shadow copy so we can also do emulation whenever a process tries to write to it. https://gist.github.com/scorpionzezz/763cd6b9f257105f2941e104cf6f2d8e

Now every time a process tries to write to either the empty page and the shadow copy, the written value will be not “stored” in the memory. Thus, it hides DRAKVUF better.

Conclusion

This project increases the stealthiness level of DRAKVUF. With a high level of stealthiness, it opens up the potential for a new generation honeypots, intrusion detection systems and dynamic malware analysis where it will be hard for the adversary to detect the presence of the monitoring system.

Acknowledgement

Thanks to Tamas K Lengyel and Varga-Perke Balint you rock! Thank you for your help and patience. Thank you also for Benjamin Taubmann for the support and of course Honeynet and Google for GSOC 2018 🙂

 

 

My GSoC Experience: Allow Setting up Shared Memory Regions between VMs from xl Config File

This blog was written by Zhongze Liu. Zhongze Liu is a student studying information security in Huazhong University of Science and Technology in Wuhan, China. He recently took part in GSoC 2017 where he worked closely with the Xen Project community on “Allowing Sharing Memory Regions between VMs from xl Config.” His interests are low-level hacking and system security (especially cloud and virtualization security).

I got to know the Xen Project about one year ago when I was working on a virtualization security project in a system security lab. It was the very first time that I received hands-on experience with a Type-I hypervisor. I was very interested in its internals and wanted to explore more of it by reading its code and performing some hacking on it. This is also what I was able to do this summer while I worked as a GSoC student with the Xen Project community. My specific focus was on setting up shared memory regions among VMs from a new xl config entry.

The purpose of this GSoC project is to allow simple guests that don’t have grant table support to be able to communicate via one or more shared memory regions. Such guests are not uncommon in the embedded world, and this project makes it possible for these poor guests to communicate with their friends.

This project involves many components of Xen, from the xl utility, xenstore, all the way down to the hypervisor itself. The implementation plan is quite straightforward: (1) during domain creation: parse the config –> map the pages –> write down in the xenstore fs what has been done;  (2) during domain destruction: read from xenstore the current status –> unmap the pages –> clean up the related xenstore entries. More details can be found in my proposal posted on the xen-devel mailing list. The tangible outcome is a patch set adding changes to xl, libxl, libxc, xsm and flask.

I met quite a few challenges during the project. The first and biggest one turned out to be how to design an appropriate syntax for the new config entry. The syntax has to be flexible and friendly to users. And the hardest part is how to control the stage-2 page permissions and cache attributes — we currently don’t have such a hypercall to control the stage-2 page attributes, but the clients are asking for the control over these attributes. I read a lot of documents about stage-2 page attributes on both x86 and ARM, and wrote a proposal for a new hypercall that would solve this issue.

After I made this proposal, I discovered that it would take up too much time to discuss the details in my proposal, not to mention implementing it. After discussing this challenge with my mentors, we decided to leave this as a TODO (see the “Future Directions” section in the project proposal), and only support the default attributes in the very first version of this project.

Next challenge: the “map the pages” step in the plan is easier said than done. After implemented the tool stack side, I moved forward to test my code, but kept getting errors on mapping the pages. By putting many printks through the whole code path, I found something blocking the way: On x86, adding foreign pages from one DomU to another by modifying p2m entries is not allowed.

Why? (1) the default xsm policy doesn’t allow this; (2) p2m tear-down is not implemented —doing so will screw up the refcount of the pages.

Fixing reason (2) is not a trivial task, but luckily, p2m tear-down is already implemented on the ARM side. So I decided to mark this new config entry as unsupported on x86, and continue to implement the ARM side. The fix to (1) turned out to be some changes to the xsm interface for xsm_map_gmfn_foreign, the dummy xsm policy, and the corresponding flask hook.

The last challenge that I’m going to talk about is testing. To test out ARM code, I followed this instruction on the Xen wiki to setup an emulator and another instruction on cross-compiling Xen and the tool stack for ARM. I’m not using Debian, so some of the handy tools provided by Debian are not available for me. I have to find alternative solutions to some of the critical steps and during my experiment, I found docker is the most distribution-independent solution which in the mean time won’t bring too much performance overhead. I created a Debian-based docker images with all the tools and dependencies required to build Xen, and every time I went to launch a build, I just needed to do a ‘docker run -v local/xen/path:docker/xen/path -it image-name build-script-name.sh‘.  I’m planning to post my Dockerfile to the Xen wiki so that others can build their own cross-building environment with a simple ‘docker build’.

I’ve really learned a lot during the process, from my mentors, and from the Xen Project community. Additionally:

  • I’ve improved my coding skills
  • I’ve learned more about the Xen Project and its internals
  • I’ve learned many efficient git tricks, which will be very useful in my future projects
  • I’ve read about memory management on ARM and x86
  • I’ve learned how to setup a rootfs, emulator, kernel, drivers and cross-compiling environment to test out ARM programs
  • And most importantly, I’ve learned how to work with an open source community.

And no words are strong enough to express my many thanks to all the people in the community who have helped me so far, especially Stefano, Julien, and Wei. They’ve been very supportive and responsive from the very beginning, giving me valuable suggestions and answering my sometimes stupid questions.

I’m very glad that I was invited to the Xen Project Summit in Budapest. It was really a great experience to meet so many interesting people there. And many thanks to Lars and Mary, who helped me in getting my visa to the event and offered me two T-shirts to help me through the hard times when my luggage was delayed.

The GSoC internship is coming to an end, and it’s just my first step to contributing to the Xen Project. I like this community and I am looking forward to contributing to it more and learning more from it.

 

My GSoC experience: Fuzzing the hypervisor

This blog post was written by Felix Schmoll, currently studying Mechanical Engineering at ETH Zurich. After obtaining a Bachelor in Computer Science from Jacobs University he spent the summer working on fuzzing the hypervisor as a Google Summer of Code student. His main interests in code are low-level endeavours and building scalable applications.

Five months ago, I had never even heard of fuzzing, but this summer, I worked on fuzzing the Xen Project hypervisor as a Google Summer of Code student.

For everybody that is not familiar with fuzzing: it is a way to test interfaces. The most primitive form of it is to repeatedly generate random input and feed it to the interface. A more advanced version is coverage-guided fuzzing, which uses information on the code path taken by the binary to permute the input further. The goal of this project was to build a prototype of fuzzing the hypercall-interface, seeing if one could make the hypervisor crash with a definite sequence of hypercalls.

American Fuzzy Lop (AFL) is by far the most popular fuzzer, and so it was chosen as the one to be run on the hypervisor. As it is a fuzzer for user-space programs, it had to be ported to the kernel. To make this work, the first step was to allow it to obtain feedback on the coverage from Xen by implementing a hypercall. Further, a mechanism was needed to execute the hypercalls from a domain other than dom0 (there are many ways to stop the hypervisor from dom0). For this purpose, an XTF-test case was instrumented to run as a server, receiving test cases from an AFL-instance. In the end, changes were made to the hypervisor, libxl, xenconsole, XTF and AFL.

The biggest challenge of all was finding my way around the code base of Xen. A lot of components were relevant to the project, and it would be unrealistic to expect anybody to read all of the code at once. While documentation was at times scarce, a helpful community of experts was always available on IRC. It was also a great experience to meet these people at the Xen Project Summit in Budapest.

The result of my summer project are numerous patches. While there were no bugs actually found (i.e. the hypervisor never crashed), valuable experience was collected for future projects. I am confident that by building up on the prototype it will be possible to improve the reliability of Xen. A first step would be to pass the addresses of valid buffers into hypercalls. For a description of more possible improvements please read the technical summary of the project.

Lastly, I would like to thank everybody involved with GSoC, Xen Project and in particular my great mentor Wei Liu for allowing me to experience how it is to work on a well-lead open-source project.

Xen Project Participates in Google Summer of Code and Outreachy

This is a quick announcement that the Xen Project is again participating in Google Summer of Code (GSoC), a program that awards three-month paid stipends to University students to work on open source projects, with the goal to get open source experience. The Xen Project will also again participate in Outreachy, which is an internship program that organises three-month paid internships with free and open-source software projects for people who are typically underrepresented in the technology industry. Outreachy has been helping women (cis and trans), trans men, genderqueer people and people from other discriminated backgrounds get involved in free and open source software for several years. The Xen Project is proud that it has participated continually in Outreachy (and its predecessor OPW) for 4 years.

I want to participate, how do I get started?

If you are not at all familiar with programs such as GSoC and Outreachy, have a quick look at our introduction. In a nutshell, both programs go through several stages:

  • Check eligibility requirements.
  • Now until application period: Preparation by working on small tasks (also called micro-tasks) within our community to identify a suitable project and to familiarise yourself with the technology.
  • Application Period (aka paperwork): For GSoC, the application system is open from March 20 to Apr 3, 2017; however you should work on micro-tasks before and prepare your application together with a mentor as early as possible. For Outreachy, the application system is already open and will close March 30, 2017 (but you can edit and modify proposals submitted in agreement with your mentor until April 28th).
  • Selection Period: After applying to participate, our mentors will chose the most promising candidates. Successful candidates will be announced on the following dates: April 28 (Outreachy), May 4 (GSoC).
  • Internship Duration: May 30 to August 29 (GSoC) and August 30 (Outreachy).

For a list of projects for participants and more information on how to apply, check our Xen Project 2017 Summer Internship Portal. We have many different projects in many different areas: from Hypervisor work, projects in Mirage OS, to tools and test related tasks. Note that we will be adding extra projects to this page in the coming weeks and that applicants can suggest projects on their own.

You may also want to check out the pages of GSoC mentoring organisations which we collaborate with. Sometimes, you will find Xen related projects there: FreeBSD (currently 2 projects), QEMU, Libvirt.

Learn about the Experience of past Participants

At a past Xen Project Developer Summit, we ran a panel discussion that included Outreachy interns, GSoC students as well as mentors.


You may also want to read Women interns rocking open source at Xen Project.

Xen Google Summer of Code – Migration from Memory Ballooing to Memory Hotplug

From xen-devel mailing list, an introduction and proposal from Daniel Kiper on his project (see email and community responses to proposal at http://lists.xensource.com/archives/html/xen-devel/2010-07/msg00410.html):

This year I put an proposal regarding migration from memory ballooning to memory hotplug in Xen to Google Summer of Code 2010 (it was one of my two proposals). It was accepted and now I happy GSoC 2010 student.
My mentor is Jeremy Fitzhardinge. I would like to thank him for his patience and supporting hand.

OK, let’s go to details. When I was playing with Xen I saw that ballooning does not give possibility to extend memory over boundary declared at the start of system. Yes, I know that is by desing however I thought that it is a limitation which could by very annoing in some enviroments (I think especially about servers). That is why I decided to develop some code which remove that one. At the beggining I thought that it should be replaced by memory hotplyg however after some test and discussion with Jeremy we decided to link balooning (for memory
removal) with memory hotplug (for extending memory above boundary declared at the startup of system). Additionaly, we decided to implement this solution for Linux Xen gustes in all forms (PV/i386,x86_64 and HVM/i386,x86_64).

Now, I have done most of the planned tests and wrote a PoC.

Short description of current algorithm (it was prepared for PoC and it will be changed to implement convenient mechanism for user):
– find free (not claimed by another memory region or device)
memory region of PAGES_PER_SECTION << PAGE_SHIFT
size in iomem_resource,
– find all PFNs for choosen memory region
(addr >> PAGE_SHIFT),
– allocate memory from hypervisor by
HYPERVISOR_memory_op(XENMEM_populate_physmap, &memory_region),
– inform system about new memory region and reserve it by
mm/memory_hotplug.c:add_memory(memory_add_physaddr_to_nid(start_addr),
start_addr, PAGES_PER_SECTION << PAGE_SHIFT),
– online memory region by
mm/memory_hotplug.c:online_pages(start_addr >> PAGE_SHIFT,
PAGES_PER_SECTION << PAGE_SHIFT).

Currently, memory is added and onlined in 128MiB blocks (section size for x86), however I am going to do that in smaller chunks.
Additionally, some things are done manually however it will be changed in final implementation.
I would like to mention that this solution does not require any change in Xen hypervisor.

I am going to send you first version of patch (fully working) next week.

Development Projects to Work On

I have spoken with several community members today at Xen Summit and they have great project ideas for Xen but not enough time to work on them. I am also aware that many university students are looking for research projects for their masters or phd work and would like to contribute to Xen at the same time. To satisfy both parties I have created a new Wiki page where people can list project ideas;  http://wiki.xensource.com/xenwiki/Xen_Work.

For community members, please list the project idea and your contact information so students can get in touch with you for more information.

For students, please consider the many projects listed and contact the community member for more information.