Using Xen Project on OpenStack “Juno” via Libvirt

By Xing Lin

This document describes steps I took to setup a compute node based on Ubuntu 14.04 for OpenStack “juno”, using the Xen Project via libvirt approach. Openstack does not support this approach well as it is in Group C of the hypervisor support matrix for Openstack. You can hardly find any tutorial online describing this approach and this might be the first. Let’s get started!


Follow “OpenStack Installation Guide for Ubuntu 14.04″ to setup the control node and network node, following the three-node architecture with OpenStack Networking (neutron). This involves lots of configuration and could take a day or two. Check that the control node and network node is working.


NOTE: Steps 3, 4, and 5 are workarounds for bugs present in Ubuntu 14.04 (and probably in other Debian derivatives of that era). Future releases of Ubuntu may not require these workarounds.  See the REFERENCES section below to review the actual bug reports which have been filed.

1. Add OpenStack juno to the repository:

apt-get update
apt-get install software-properties-common
add-apt-repository cloud-archive:juno
apt-get update

2. Install nova-compute-xen, sysfsutils and python nova client:

apt-get install nova-compute-xen sysfsutils python-novaclient

3. Install qemu-2.0.2 with a patch fixing unmapping of persistent grants. Current qemu releases (including 2.0.2, 2.1.2 and 2.2.0-rc1) do not have this patch included and this will result in Dom0 kernel crashes when creating a Xen Project DomU from OpenStack GUI (horizon). I have applied this patch and made the modified qemu available in github:

cd qemu-2.0.2-master/
apt-get build-dep qemu
make -j16
make install

4. Add a patch to /etc/init.d/xen to start qemu process during startup:

--- /etc/init.d/xen	2014-11-18 20:54:10.788457049 -0700
+++ /etc/init.d/xen.bak	2014-11-18 20:53:14.804107463 -0700
@@ -228,6 +228,9 @@ case "$1" in
		*) log_end_msg 1; exit ;;
	log_end_msg 0
+	/usr/local/bin/qemu-system-i386 -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize \
+	    -monitor /dev/null -serial /dev/null -parallel /dev/null \
+	    -pidfile /var/run/

5. Create a link at /usr/bin for pygrub:

 ln -s /usr/lib/xen-4.4/bin/pygrub /usr/bin/pygrub

6. Reboot the machine and boot into Xen Project Dom0.

7. Edit the /etc/nova/nova.conf, to configure nova service. You can also follow steps in the OpenStack installation guide to configure nova as a compute node.

  • In the default section, add the following:
 rpc_backend = rabbit
 rabbit_host = controller
 rabbit_password = RABBIT_PASS
 auth_strategy = keystone
 vnc_enabled = True
 vncserver_listen =
 vncserver_proxyclient_address = MANAGEMENT_INTERFACE_IP_ADDRESS
 novncproxy_base_url = http://controller:6080/vnc_auto.html
 verbose = True

MANAGEMENT_INTERFACE_IP_ADDRESS is the IP address of the management network interface for this compute node, typically for the first compute node.

  • Add keystone_authtoken section:
 auth_uri = http://controller:5000/v2.0
 identity_uri = http://controller:35357
 admin_tenant_name = service
 admin_user = nova
 admin_password = NOVA_PASS
  • Add glance section:

8. Verify the content of /etc/nova/nova-compute.conf is as follows:


9. Install and configure network component in compute node. Follow the steps outlined in this OpenStack install guide for neutron compute nodes. Note, in /etc/neutron/neutron.conf, I did not set “allow_overlapping_ips = True” in the default section, because it is said to set this property to False if both Neutron and the nova security groups are used together.

10. Final step: now you should be able to launch an instance from horizon. In my case, I launched an instance running cirros-0.3.3-x86_64. When I login to the compute node, I can see this instance running with virsh:

# virsh --connect=xen:///
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # list
Id    Name                           State
1     instance-0000003b              running


About the author:
Xing is a PhD student in the School of Computing at the University of Utah. His primary interests are in file and storage systems. He has built a NFS client driver for Hadoop (will open source soon) and proposed a new data transformation for improving compression, called “migratory compression” (check out our USENIX FAST ’14 paper).

5 thoughts on “Using Xen Project on OpenStack “Juno” via Libvirt

  1. bjso


    An error has occurred. since the 3-Section. (3. Install qemu-2.0.2 with a patch…)
    Currently, the following states.

    I don’t know what to do about this.
    How can we solve the problem?

    # cat /etc/issue
    Ubuntu 14.04.1 LTS \n \l

    # xen info
    host : xen-test
    release : 3.13.0-43-generic
    version : #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014
    machine : x86_64
    nr_cpus : 32
    max_cpu_id : 31
    nr_nodes : 2
    cores_per_socket : 8
    threads_per_core : 2
    cpu_mhz : 2100
    hw_caps : bfebfbff:2c100800:00000000:00003f00:17bee3ff:00000000:00000001:00000000
    virt_caps : hvm hvm_directio
    total_memory : 131026
    free_memory : 127
    sharing_freed_memory : 0
    sharing_used_memory : 0
    outstanding_claims : 0
    free_cpus : 0
    xen_major : 4
    xen_minor : 4
    xen_extra : .1
    xen_version : 4.4.1
    xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
    xen_scheduler : credit
    xen_pagesize : 4096
    platform_params : virt_start=0xffff800000000000
    xen_changeset :
    xen_commandline : placeholder
    cc_compiler : gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2
    cc_compile_by : stefan.bader
    cc_compile_domain :
    cc_compile_date : Wed Nov 26 14:12:33 UTC 2014
    xend_config_format : 4

    # nova-manage version

    # virsh –connect=xen:///
    error: failed to connect to the hypervisor
    error: internal error: libxenlight state driver is not active

    # service nova-compute start
    nova-compute start/running, process 12082

    # service nova-compute status
    nova-compute stop/waiting

    # nova-manage service list
    Binary Host Zone Status State Updated_At
    nova-cert xen-test internal enabled :-) 2014-12-23 10:50:29
    nova-consoleauth xen-test internal enabled :-) 2014-12-23 10:50:29
    nova-scheduler xen-test internal enabled :-) 2014-12-23 10:50:28
    nova-conductor xen-test internal enabled :-) 2014-12-23 10:50:25
    nova-compute xen-test nova disabled XXX 2014-12-23 05:23:19

    # cat /var/log/nova/nova-compute.log
    INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
    INFO nova.virt.driver [-] Loading compute driver ‘libvirt.LibvirtDriver’
    INFO oslo.messaging._drivers.impl_rabbit [req-3a42a00c-0fdf-4dbd-9b80-0984214d604f ] Connecting to AMQP server on controller:5672
    INFO oslo.messaging._drivers.impl_rabbit [req-3a42a00c-0fdf-4dbd-9b80-0984214d604f ] Connected to AMQP server on controller:5672
    INFO oslo.messaging._drivers.impl_rabbit [req-3a42a00c-0fdf-4dbd-9b80-0984214d604f ] Connecting to AMQP server on controller:5672
    INFO oslo.messaging._drivers.impl_rabbit [req-3a42a00c-0fdf-4dbd-9b80-0984214d604f ] Connected to AMQP server on controller:5672
    AUDIT nova.service [-] Starting compute node (version 2014.2)
    ERROR nova.virt.libvirt.driver [-] Connection to libvirt failed: internal error: libxenlight state driver is not active
    TRACE nova.virt.libvirt.driver Traceback (most recent call last):
    TRACE nova.virt.libvirt.driver File “/usr/lib/python2.7/dist-packages/nova/virt/libvirt/”, line 839, in _connect
    TRACE nova.virt.libvirt.driver libvirt.openAuth, uri, auth, flags)
    TRACE nova.virt.libvirt.driver File “/usr/lib/python2.7/dist-packages/eventlet/”, line 139, in proxy_call
    TRACE nova.virt.libvirt.driver rv = execute(f,*args,**kwargs)
    TRACE nova.virt.libvirt.driver File “/usr/lib/python2.7/dist-packages/eventlet/”, line 77, in tworker
    TRACE nova.virt.libvirt.driver rv = meth(*args,**kwargs)
    TRACE nova.virt.libvirt.driver File “/usr/lib/python2.7/dist-packages/”, line 105, in openAuth
    TRACE nova.virt.libvirt.driver if ret is None:raise libvirtError(‘virConnectOpenAuth() failed’)
    TRACE nova.virt.libvirt.driver libvirtError: internal error: libxenlight state driver is not active
    TRACE nova.virt.libvirt.driver
    ERROR nova.openstack.common.threadgroup [-] Connection to the hypervisor is broken on host: xen-test
    TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
    TRACE nova.openstack.common.threadgroup File “/usr/lib/python2.7/dist-packages/nova/openstack/common/”, line 125, in wait
    TRACE nova.openstack.common.threadgroup x.wait()
    TRACE nova.openstack.common.threadgroup File “/usr/lib/python2.7/dist-packages/nova/openstack/common/”, line 47, in wait
    TRACE nova.openstack.common.threadgroup return self.thread.wait()
    TRACE nova.openstack.common.threadgroup File “/usr/lib/python2.7/dist-packages/eventlet/”, line 168, in wait
    TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
    TRACE nova.openstack.common.threadgroup File “/usr/lib/python2.7/dist-packages/eventlet/”, line 116, in wait
    TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
    TRACE nova.openstack.common.threadgroup File “/usr/lib/python2.7/dist-packages/eventlet/hubs/”, line 187, in switch
    TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
    TRACE nova.openstack.common.threadgroup File “/usr/lib/python2.7/dist-packages/eventlet/”, line 194, in main
    TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
    TRACE nova.openstack.common.threadgroup File “/usr/lib/python2.7/dist-packages/nova/openstack/common/”, line 490, in run_service
    TRACE nova.openstack.common.threadgroup service.start()
    TRACE nova.openstack.common.threadgroup File “/usr/lib/python2.7/dist-packages/nova/”, line 164, in start
    TRACE nova.openstack.common.threadgroup self.manager.init_host()
    TRACE nova.openstack.common.threadgroup File “/usr/lib/python2.7/dist-packages/nova/compute/”, line 1125, in init_host
    TRACE nova.openstack.common.threadgroup self.driver.init_host(
    TRACE nova.openstack.common.threadgroup File “/usr/lib/python2.7/dist-packages/nova/virt/libvirt/”, line 696, in init_host
    TRACE nova.openstack.common.threadgroup self._do_quality_warnings()
    TRACE nova.openstack.common.threadgroup File “/usr/lib/python2.7/dist-packages/nova/virt/libvirt/”, line 679, in _do_quality_warnings
    TRACE nova.openstack.common.threadgroup caps = self._get_host_capabilities()
    TRACE nova.openstack.common.threadgroup File “/usr/lib/python2.7/dist-packages/nova/virt/libvirt/”, line 3299, in _get_host_capabilities
    TRACE nova.openstack.common.threadgroup xmlstr = self._conn.getCapabilities()
    TRACE nova.openstack.common.threadgroup File “/usr/lib/python2.7/dist-packages/nova/virt/libvirt/”, line 775, in _get_connection
    TRACE nova.openstack.common.threadgroup wrapped_conn = self._get_new_connection()
    TRACE nova.openstack.common.threadgroup File “/usr/lib/python2.7/dist-packages/nova/virt/libvirt/”, line 728, in _get_new_connection
    TRACE nova.openstack.common.threadgroup wrapped_conn = self._connect(self.uri(), self.read_only)
    TRACE nova.openstack.common.threadgroup File “/usr/lib/python2.7/dist-packages/nova/virt/libvirt/”, line 848, in _connect
    TRACE nova.openstack.common.threadgroup raise exception.HypervisorUnavailable(
    TRACE nova.openstack.common.threadgroup HypervisorUnavailable: Connection to the hypervisor is broken on host: xen-test
    TRACE nova.openstack.common.threadgroup

      1. bjso

        Thank you very much.
        The error has been resolved to your advice. Wow.

        Happy New Year~~.

        # virsh –connect=xen:///
        Welcome to virsh, the virtualization interactive terminal.
        Type: ‘help’ for help with commands
        ‘quit’ to quit
        virsh # list
        Id Name State
        1 instance-00000006 running

  2. idono1301


    I have been trying to get this working, except when running all the services on a single machine. I am
    unable to boot any guests through both horizon nor nova CLI. When looking at the logs in
    /var/log/libvirt/libxl/instance-XXXXXXX.log, I see the following errors:

    libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge online [-1] exited with error status 1
    libxl: error: libxl_device.c:1085:device_hotplug_child_death_cb: script: ip link set vif1.0 name tap67df6826-05 failed
    libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x7f3a78009870: deregister unregistered
    libxl: error: libxl_create.c:1226:domcreate_attach_vtpms: unable to add nic devices

    From searching on google, there are few reports of this, but no solution.

    Could you please assist?

    Thank you!

    1. Russell Pavlicek Post author

      Could it be something similar to this report (which is older, different version, but seems to have some similar attributes)?

      If so, that thread ends like this:

      ‘I had iproute2 3.8.0 installed, but I also have the global “minimal”
      use flag, with which the iproute2 ebuild only installs tc.
      Ironically, the iproute2 package was the only installed package to be
      modified by the minimal use flag, so it only saved me
      something I actually needed.’

      If this doesn’t help, I suggest you put your question in the xen-users mailing list. That’s the best way to get help debugging a situation.


Leave a Reply