This is a repost of a tutorial published initially on community.arm.com – Thank you to Andrew Wafaa for allowing us to repost.
With ARM entering the server space, a key technology in play in this segment is Virtualization. Virtualization is not a tool solely for servers and the data center, it is also used in the embedded space in segments like automotive and it is also starting to be used in mobile.
This is not a new technology, IBM pioneered it in the 1960s, and there are many different hypervisors implementing different methods of virtualization. In the Open Source realm there are two major hypervisors: KVM and Xen. Both are interact directly with the Linux kernel, however KVM is solely in the Linux domain whereas Xen works with Linux, *BSD and other UNIX variants.
In the past it was generally accepted that there are two types of hypervisor, Type1 (also known as bare metal or native) where the hypervisor runs directly on the host server and controls all aspects of the hardware and manages the guest operating systems, and Type2 (also known as hosted) where the hypervisor runs within a normal operating system; under this classification Xen falls into the Type1 camp and KVM fell into the Type2 camp. However the modern implementations of the hypervisors has now blurred the lines of distinction.
This time round I’ll be taking a look at the Xen Hypervisor, which is now one of the Linux Foundation’s collaborative projects. Here is a brief overview of some of Xen’s features:
- Small footprint. Based on a microkernel design, it has a limited interface to the guest virtual machine and takes up around 1MB of memory.
- Operating system agnostic. Xen works well with BSD variants and other UNIX systems, although most deployments use Linux.
- Driver Isolation. This allows the main device drivers of a system to run inside the VM, which enables the VM containing drivers to be rebooted in the event of a crash or compromise without affecting the host or other guests. In the Xen model the majority of the device drivers run in virtual machines rather than in the hypervisor, as well as allowing reusing of existing OS driver stacks this allows the VM containing the driver to be rebooted without affecting the host of other guests. Individual drivers can even be run in separate VMs in order to improve isolation and fault tolerance or just to take advantage of differing OS functionality.
- Paravirtualization (PV). This style of port enables Xen to run on hardware that doesn’t have virtualization extensions, such as Cortex-A5/A8/A9 in ARM’s case. There can also be some performance gains for some PV guests, but this requires the guests to be modified and prevents “out of the box†implementations of operating systems.
- No emulation, no QEMU. Emulated interfaces are slow and insecure. By using hardware virtualization extensions and IO paravirtualization, Xen removes any need for emulation. As a result you have a smaller code base and better performances
The Xen hypervisor runs directly on the hardware and is responsible for handling CPU, Memory, and interrupts. It is the first program running after exiting the bootloader. Virtual machines then run atop of Xen. A running instance of a virtual machine in Xen is called a DomU or guest. The controller for the guest VMs is a special host VM called Dom0 and contains the drivers for all the devices in the system. Dom0 also contains a control stack to manage virtual machine creation, destruction, and configuration.
Pieces of the puzzle:
- The Xen Hypervisor is a lean software layer that runs directly on the hardware and as mentioned is responsible for managing CPU, memory, and interrupts. The hypervisor itself has no knowledge of I/O functions such as networking and storage.
- Guest Domains/Virtual Machines (DomU)Â are virtualized environments, each running their own operating system and applications. On other architectures Xen supports two different virtualization modes: Paravirtualization (PV) and Hardware-assisted or Full Virtualization (HVM). Both guest types can be used at the same time on a single Xen system. On ARM there is only one mode for virtualization that is more of a hybrid of the two, which effectively is hardware based with paravirt extensions. Some call this mode PVHVM. Xen guests are totally isolated from the hardware and have no privilege to access hardware or I/O functionality. Which is where DomU comes in, standing for unprivileged domain.
- The Control Domain (or Dom0) is a specialized Virtual Machine that has special privileges like the capability to access the hardware directly, handles all access to the system’s I/O functions and interacts with the other Virtual Machines. It also exposes a control interface to the outside world, through which the system is controlled. The Xen hypervisor will not function without Dom0, which is the first VM started by the system.
- Toolstack and Console: Dom0 contains a control stack (known as Toolstack) that allows a user to manage virtual machine creation, destruction, and configuration. The toolstack exposes an interface that is either driven by a command line console, by a graphical interface or by a cloud orchestration stack (OpenStack or CloudStack.)
- Xen-enabled operating systems:Â Running an operating system as Dom0 or DomU requires the operating system kernel to be Xen enabled. However porting an operating system to Xen on ARM is simple: it just needs a few new drivers for the Xen paravirtualized IO interfaces. Existing open source PV drivers in Linux and FreeBSD are likely to be reusable. Linux distributions that are based on a recent Linux kernel (3.8+) are already Xen enabled and usually contain packages for the Xen hypervisor and tools.
The latest version of Xen is 4.4.0, which was released in March and has support for both ARMv7 and ARMv8. For this exercise I’ll be looking at using Xen on ARMv8 with the Foundation Model.
Please consult the Xen Wiki for more information on using Xen with Virtualization Extensions and using Xen with Models. For discussion/review/information and help there are mailinglists and IRC.
Development set up:
You can use whichever Linux distribution you prefer, so long as you have a suitable cross-compilation environment set up. I’m using openSUSE 13.1 with the Linaro Cross Toolchain for AArch64.
Typographic explanation:
host$Â = run as a regular user on host machine
host#Â = run as root user on host machine (can use sudo if you prefer)
chroot>Â = run as root user in chroot environment
model>Â = run as root user in a running Foundation Model
The first steps are to build Xen and a Linux kernel for use in both Dom0 and DomU machines. We then package Xen and Linux along with a Device Tree together for Dom0 to be used in the model using boot-wrapper.
Build Xen:
If using Linaro’s toolchain, ensure the /bin directory is in your $PATH
host$ git clone git://xenbits.xen.org/xen.git xen
host$ cd xen
host$ git checkout RELEASE-4.4.0
There is a small build bug due to use of older autotools which will be fixed in the 4.4.1 release. Rather than wait for the next release, we’ll just backport it now.
host$ git cherry-pick 0c68ddf3085b90d72b7d3b6affd1fe8fa16eb6be
There is also a small bug in GCC with PSR_MODE see bug LP# 1169164. Download the attached PSR_MODE_workaround.patch
host$ patch -i PSR_MODE_workaround.patch -p1
host$ make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- CONFIG_EARLY_PRINT=fastmodel
host$ cd ..
Build Linux:
host$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
host$ cd linux
host$ git checkout v3.13
Create a new kernel config:
host$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
host$ sed -e 's/.*CONFIG_XEN is not set/CONFIG_XEN=y/g' -i .config
host$ sed -e 's/.*CONFIG_BLK_DEV_LOOP is not set/CONFIG_BLK_DEV_LOOP=y/g' -i .config
host$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- oldconfig
Make sure to select Y to all Xen config options
I have attached a kernel.config which has all the required options enabled for reference.
host$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image
host$ cd ..
Obtain the Foundation Model:
In a browser go to http://www.arm.com/products/tools/models/fast-models/foundation-model.php
Scroll to the bottom and select “Download Nowâ€
This should provide FM000-KT-00035-r0p8-52rel06.tgz
Extract the tarball
host$ tar xaf FM000-KT-00035-r0p8-52rel06.tgz
Build Boot Wrapper and device tree.
It is common to run the models without real firmware. In this case a boot-wrapper is needed to provide a suitable boot time environment for Xen, which allows booting into Non-Secure HYP mode and providing boot modules etc.:
host$ git clone -b xen-arm64 git://xenbits.xen.org/people/ianc/boot-wrapper-aarch64.git
host$ cd boot-wrapper-aarch64
host$ ln -s ../xen/xen/xen Xen
host$ ln -s ../linux/arch/arm64/boot/Image Image
Use the attached foundation-v8.dts to build the device tree blob
host$ dtc -O dtb -o fdt.dtb foundation-v8.dts
host$ make CROSS_COMPILE=aarch64-linux-gnu- FDT_SRC=foundation-v8.dts IMAGE=xen-system.axf
host$ cd ..
Run the Model to make sure the kernel functions, it will panic as we haven’t setup the rootfs yet:
host$ ./Foundation_v8pkg/models/Linux64_GCC-4.1/Foundation_v8 \
--image boot-wrapper-aarch64/xen-system.axf
Chroot Build Environment
Next we create a suitable chroot build environment using the AArch64 port of openSUSE. We will use the qemu-user-static support for AArch64 to run the chroot on the (x86) host.
First we build the qemu binary, then construct the chroot, finally we build Xen in the chroot environment.
Building qemu-aarch64-user
host$ git clone https://github.com/openSUSE/qemu.git qemu-aarch64
host$ cd qemu-aarch64
host$ git checkout aarch64-work
Install some build dependencies:
host# zypper in glib2-devel-static glibc-devel-static libattr-devel-static libpixman-1-0-devel ncurses-devel pcre-devel-static zlib-devel-static
host$ ./configure --enable-linux-user --target-list=arm64-linux-user --disable-werror --static
host$ make -j4
host$ ldd ./arm64-linux-user/qemu-arm64
not a dynamic executable
This last step is to verify that the resulting binary is indeed a static binary. We will copy it into the chroot later on.
We now need to enlighten binfmt misc about aarch64 binaries:
On openSUSE:
host# cp scripts/qemu-binfmt-conf.sh /usr/sbin/
host# chmod +x /usr/sbin/qemu-binfmt-conf.sh
host# qemu-binfmt-conf.sh
On Debian:
host# update-binfmts --install aarch64 /usr/bin/qemu-aarch64-static \
--magic '\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7' \
--mask '\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff'
host$ cd ..
Build the chroot environment
host$ wget http://download.opensuse.org/ports/aarch64/distribution/13.1/appliances/openSUSE-13.1-ARM-JeOS.aarch64-rootfs.aarch64-1.12.1-Build32.2.tbz
Note: the file name may change due to the continuous image building, if the above does not work verify the latest version of the tarball here.
host$ mkdir aarch64-chroot
host# tar -C aarch64-chroot -xaf openSUSE-12.3-AArch64-JeOS-rootfs.aarch64-1.12.1-Build5.13.tbz
Install the qemu binary into the chroot environment
host# cp qemu-aarch64/arm64-linux-user/qemu-arm64 aarch64-chroot/usr/bin/qemu-aarch64-static
host# cp /etc/resolv.conf aarch64-chroot/etc/resolv.conf
Build the Xen tools in the chroot environment (finally)
Copy the Xen sources into the chroot
host# cp -r xen aarch64-chroot/root/xen
Chroot into the aarch64 environment
host# chroot aarch64-chroot /bin/sh
We now need to install some build dependencies
chroot> zypper install gcc make patterns-openSUSE-devel_basis git vim libyajl-devel python-devel wget libfdt1-devel libopenssl-devel
If prompted to trust the key, I’ll let you choose whether to trust permanently or just this time (personally I chose to always trust the key).
chroot> cd /root/xen
chroot> ./configure
chroot> make dist-tools
chroot> exit
The Xen tools are now in aarch64-chroot/root/xen/dist/install
Root filesystem and image:
We will create an ext3 formatted filesystem image, we will also use a simplified initscript to avoid long waits while running the model.
host$ wget http://download.opensuse.org/ports/aarch64/distribution/13.1/appliances/openSUSE-13.1-ARM-JeOS.aarch64-rootfs.aarch64-1.12.1-Build32.2.tbz
This is the same rootfs tarball as use for the chroot. You can re-use the previously downloaded tarball if you wish.
host$ dd if=/dev/zero bs=1M count=1024 of=rootfs.img
host$ /sbin/mkfs.ext3 rootfs.img
Say yes, we know it’s not a block device
host# mount -o loop rootfs.img /mnt
host# tar -C /mnt -xaf openSUSE-13.1-ARM-JeOS.aarch64-rootfs.aarch64-1.12.1-Build32.2.tbz
Install the Xen tools that we built earlier
host# rsync -aH aarch64-chroot/root/xen/dist/install/ /mnt/
Create the init scipt:
host# cat > /mnt/root/init.sh <<EOF
#!/bin/sh set -x mount -o remount,rw / mount -t proc none /proc mount -t sysfs none /sys mount -t tmpfs none /run mkdir /run/lock mount -t devtmpfs dev /dev /sbin/udevd --daemon udevadm trigger --action=add mkdir /dev/pts mount -t devpts none /dev/pts mknod -m 640 /dev/xconsole p chown root:adm /dev/xconsole /sbin/klogd -c 1 -x /usr/sbin/syslogd cd /root export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin exec /bin/bash EOF
host# chmod +x /mnt/root/init.sh
Get missing runtime dependencies for Xen
host$ wget http://download.opensuse.org/ports/aarch64/distribution/13.1/repo/oss/suse/aarch64/libyajl2-2.0.1-14.1.2.aarch64.rpm
host$ wget http://download.opensuse.org/ports/aarch64/distribution/13.1/repo/oss/suse/aarch64/libfdt1-1.4.0-2.1.3.aarch64.rpm
host# cp libyajl2-2.0.1-14.1.2.aarch64.rpm libfdt1-1.4.0-2.1.3.aarch64.rpm /mnt/root/
host# umount /mnt
Start the model
host$ ./Foundation_v8pkg/models/Linux64_GCC-4.1/Foundation_v8 \
--image boot-wrapper-aarch64/xen-system.axf \
--block-device rootfs.img \
--network=nat
Silence some of the harmless warnings
model> mkdir /lib/modules/$(uname -r)
model> depmod -a
Install the runtime dependencies:
model> rpm -ivh libfdt1-1.3.0-9.1.1.aarch64.rpm libyajl2-2.0.1-12.1.1.aarch64.rpm
model> ldconfig
Start the Xen daemon, you can ignore the harmless message about i386 qemu if it appears.
model> /etc/init.d/xencommons start
If you get an error of missing file for /etc/init.d/xencommons re-run ldconfig.
Confirm that Dom0 is up:
model> xl list
ID Mem VCPUs     State  Time(s) Domain-0 0  512    2 r-----  13.9
Congratulations, you now have a working Xen toolstack. You can now shut down the model for now.
Creation of a DomU guest
For the guest rootfs we will use a smaller OpenEmbedded based Linaro image rather than a full openSUSE image purely for space constraints.
host$ wget http://releases.linaro.org/latest/openembedded/aarch64/linaro-image-minimal-genericarmv8-20140223-649.rootfs.tar.gz
host$ dd if=/dev/zero bs=1M count=128 of=domU.img
host$ /sbin/mkfs.ext3 domU.img
Again say yes, we know it’s not a block device
host# mount -o loop domU.img /mnt
host# tar -C /mnt -xaf linaro-image-minimal-genericarmv8-20140223-649.rootfs.tar.gz
host# umount /mnt
Make the DomU rootfs and kernel available to the Dom0
host# mount -o loop rootfs.img /mnt
host# cp domU.img /mnt/root/domU.img
host# cp linux/arch/arm64/boot/Image /mnt/root/Image
Create the config for the guest
host# cat > /mnt/root/domU.cfg <<EOF
kernel = "/root/Image" name = "guest" memory = 512 vcpus = 1 extra = "console=hvc0 root=/dev/xvda ro" disk = [ 'phy:/dev/loop0,xvda,w' ] EOF
host# umount /mnt
Start the model again:
host$ ./Foundation_v8pkg/models/Linux64_GCC-4.1/Foundation_v8 \
--image boot-wrapper-aarch64/xen-system.axf \
--block-device rootfs.img \
--network=nat
model> losetup /dev/loop0 domU.img
model> /etc/init.d/xencommons start
Create the DomU using the config
model> xl create domU.cfg
View the guest’s info on the Xen console
model> xl list
Screenshot of the Dom0 host
Start the Dom0
model> xl console guest
Screenshot of the DomU guest
Now all that’s left is to have a lot of fun!
