Monthly Archives: October 2009

debugging on xen

Xen greatly facilitates debugging of OS kernels and modules by allowing
source level debug. Allow me to introduce debuggers I wrote up late 2007,
mostly from scratch, for xen. The first is gdbsx which was just recently
merged into xen unstable branch. The second is kdb, whose primary goal
is to allow debug of the hypervisor and also dom0.  kdb is somewhat
a misnomer. I started with porting kdb from linux but later found it
easier and quicker to just write one up from scratch. However, I credit
linux kdb with allowing me to learn how kernel debuggers work.

gdbsx is intended for debugging non privileged guests. It runs on
dom0 and communicates to the hypervisor directly. It is fairly light
weight. By adding little support in xen for external debuggers, I was
able to keep it fairly simple. Once started on dom0, it talks to remote
gdb and allows for standard debugging of any guest, PV or HVM, 32 or 64
bits. It resides in tools/debugger/gdbsx directory, and the README gives
details on usage. It’s as simple as building and installing a hypervisor
with gdbsx support, and then just running gdbsx on dom0.  When attached to
guest, it pauses the guest, does initial setup and is ready for debug. It
accepts commands from remote gdb, and liaisons between the remote gdb and

kdb is an assembly level debugger intended for debug of the hypervisor
and dom0, but can also be used to debug non-priviliged guests also. It finds
the basic hypervisor symbols like function names in it, and shows
meaningful stacks etc. By providing some hints on basic symbol table
for guests, it can serve to be an assembly level debugger for guests
also. kdb provides fairly large variety of commands, from standard
commands to being able to examine domain/vcpu structs, get info on
pages, looking up into m2p/p2m tables, etc…  It current resides in an
external tree at The
README in the xen/kdb sub directory provides details on usage.

I hope this helps engineers with their development and support of all of non-user level software on xen. Feedback always welcome, please post to xen-devel list.

Mukesh Rathor

The Book of Xen – A Practical Guide for the System Administrator

Here are more details on the new Xen book from No Starch Press by by Chris Takemura and Luke S. Crawford.

Full Book Promotional Page – Here
Read a Sample from the Book – Chapter 7

From Book Promotional Page ->

Xen, the open source virtualization tool, is a system administrator’s dream. Xen is a free, high-performance virtual machine monitor that lets you consolidate your hardware and finally put those unused cycles to use—without sacrificing reliability, performance, or scalability.

The Book of Xen explains everything you need to know in order to use Xen effectively, including installation, networking, memory management, and virtualized storage. You’ll also learn how to use Xen and standard Linux tools to take snapshot backups, perform QoS operations on network traffic, and limit over-aggressive disk users.

Authors Chris Takemura and Luke S. Crawford show you how to:

  • Provide virtual hosting for dozens of users, each with their own individual needs
  • Install and manage multiple guests, including various flavors of Linux, NetBSD, Solaris, and Windows
  • Choose the right virtual storage options for your needs
  • Migrate your systems seamlessly and create new images
  • Tune and benchmark your systems to make them as fast as possible
  • Troubleshoot Xen’s most common problems like network and memory management

Expert advice is priceless when it comes to running a complicated open source virtualization technology like Xen. You’ll get the advice you need in The Book of Xen.

Chris Takemura is a longtime *nix sysadmin, Perl hacker, and technological hobbyist. He’s been using Xen from its early days, and helped to build’s reputation as “hosting for the technically adept.”

Luke S. Crawford has used virtualization in production since before it was cool, virtualizing hundreds of servers for large and small corporations. He launched the VPS service in 2005, selling virtual servers based on FreeBSD jails before switching to Xen in an effort to more fairly allocate resources.

Avaya Introduces Virtualized Unified Communications Solution

More details on the Avaya product announcement:

* Complete Press Release.
* Video Overview: Avaya Auraâ„¢ System Platform Technology
* Podcast: Avaya Auraâ„¢- Innovation for Midsize Enterprises
* Requires Registration – IDC Whitepaper: Virtualizing UC: Reaping the Benefits and Understanding the Issues for Real-Time Communications

Maureen O’ Gara article on Xen aspect of the solution –
Avaya Uses Xen for Unified Communications

Things you may have missed while I was out…

Some exciting Xen news that occurred during my vacation…

I will be posting more information on these items this week and will also be posting information about the Latinoware conference in Brazil that I am speaking at this week. Hope you had a good week last week while I enjoyed the lovely rainy weather in northern Virginia.