Sunday, November 10, 2013

Missing USB devices in KVM or QEMU with Ubuntu Saucy?

I've been a long time user of Fitibits. These are small devices you wear on your person to track activity. For instance, the Fitibit Ultra can track steps taken and stairs taken among other things. The device integrates with an application and has a nice webapp to display your history.

Setup on the Ultra was done with a Windows machine as there is (well was) no Linux client. I borrowed my wife's machine (as she already had the software installed as she too had an Ultra.) Once the device was setup, some guys at Canonical created a library for Ubuntu that would allow you to sync the device properly from Ubuntu (without running Windows.) Windows was only needed at setup time.

Fitbit continued to develop new monitors and I recently ordered the Fitbit Force. The Force is essentially a smartwatch. It has a superset of features of the Ultra. It also requires setup via Windows (or Mac) and I still don't own one of those. I figured I'd just use KVM and a Windows guest image.

I'm running Ubuntu Saucy with

libvirt-bin 1.1.1-0ubuntu8
qemu-system 1.5.0+dfsg-3ubuntu5

these versions of libvirt and qemu.

Fitbit Force comes with a small dongle you plug into your usb port. (Essentially, this is a bluetooth 4 device but it is single purposed, pairing only with the Fitbit Force.) 

So I added the USB device info to my virtual machine description (using the GUI). This allows for specific usb hub and device passthrough. However, the device was never showing up in Windows (as near as I could tell.)  I spent hours debugging this, changing perms on the usb device tree, running as root, etc, to no avail.

To troubleshoot, I used a USB device lister, USBDeview, from Nirsoft. It's freeware (free as in beer). It did a nice job of listing my past and present USB devices by walking the USB tree and the registry entries of past devices. Nope, no Fitbit Force.

A bit of googling and debugging led me to this gem:
"Apparmor blocks usb devices in libvirt in Saucy" bug #1245251. By using the work around in comment #1, my Fitbit Force (and any other USB device I requested) were now available in the guest.

(I had previously tried the apparmor settings mentioned in the Managing KVM page, to no avail. Those instructions pre-date Saucy.)

Hopefully by the time anyone stumbles on this particular post, this bug will already be fixed in Saucy and Trusty, but I like to publish these lessons learned Just In Case.

Oh, and one other note, once the initial setup is done, you can pair your smartwatch to your android phone... once they add the type of android phone you have. The Nexus 5 is apparently slightly too new..... :( so I'll be using that Windows image a bit more than I planned.

Monday, August 5, 2013

Using kdump on Ubuntu in Azure

This is another of my occasional posts that may help the next guy. I call them YAHTNG, yet another helping the next guy,t blog entry...


kdump is a tool that allows you to capture (in a file) the linux kernel state when it crashes (oops). It uses the kexec functionality that's long been part of the linux kernel (since 2004 if memory serves.) In order to use this on linux, you install the linux-crashdump metapackage that in turn depends on the right bits and pieces.

apt-get install linux-crashdump

On different versions of Linux, different bits and pieces get installed. Prior to Raring, 13.04, you get one set of packages and Raring and newer, you get a different set. In either case, on Microsoft's Azure cloud and elsewhere under the hyper-v hypervisor, you will get a hang if  you just install the linux-crashdump package and then experience a crash. This is due to some Azure-specific kernel modules that get loaded in the kexec/kdump kernel. You need to exclude these modules, i.e., blacklist them. Here's how.

Older Ubuntu Releases including Precise

In 12.04 (Precise) and 12.10 (Quantal) you want to edit /etc/init.d/kdump (this is the script that runs at boot time to configure the kdump kernel. The kdump kernel gets loaded into memory and configured via this script.)

--- /etc/init.d/kdump 2013-06-28 00:09:22.400504335 +0000
+++ kdump.nohyperv 2013-06-28 00:16:48.903733116 +0000
@@ -48,6 +48,7 @@ do_start () {
  # Append kdump_needed for initramfs to know what to do, and add
  # maxcpus=1 to keep things sane.
  APPEND="$APPEND kdump_needed maxcpus=1 irqpoll reset_devices"
+ APPEND="$APPEND ata_piix.prefer_ms_hyperv=0 modprobe.blacklist=hv_vmbus,hv_storvsc,hv_utils,hv_netvsc,hid_hyperv"

  # --elf32-core-headers is needed for 32-bit systems (ok
  # for 64-bit ones too).

As you can see, we are simply prohibiting the Azure kernel modules hv_vmbus, hv_storvsc, hv_utils, hv_netvsc, and hid_hyperv from loading in the kdump kernel. They still get loaded in the regular Azure kernel (and you will want to keep them there for performance and behavior reasons.) However, if they load in the kdump kernel, they won't actually work and will "hang" the kdump kernel while they try and connect to the Azure services (or hyper-v services.) Additionally, we  prefer NOT to load the hyper-v module setting for ata_piix by setting it to zero.

After you modify this init script, you will want to reboot. (But take note and read the last section on the crashkernel as you will likely want to make that change as well, prior to rebooting.

Newer Ubuntu Release (Raring and the upcoming Saucy)

The newest releases of Ubuntu include an additional package that handles kdump configuration called kdump-tools. This package manages the kernel modules in a simple config file /etc/default/kdump-tools. You can edit that file to blacklist the appropriate modules:

    67 #KDUMP_CMDLINE_APPEND="irqpoll maxcpus=1 nousb"
    68 KDUMP_CMDLINE_APPEND="irqpoll maxcpus=1 nousb  ata_piix.prefer_ms_hyperv=0 modprobe.blacklist=hv_vmbus,hv_storvsc,hv_utils,hv_netvsc,hid_hyperv "

In addition to preferring to NOT use the ata_piix for hyperv, it also blacklists the same kernel modules as previously mentioned.

Smaller Images

Low memory (extrasmall, small) Azure instances (well, really any small images including small physical machines) unfortunately run into bug #1206691, default crashkernel setting rarely works on a system with little memory. You will need to modify /etc/grub.d/10_linux and set the crashkernel to 128M for any size instance. Do this by simply altering the range here:

   74 # add crashkernel option if we have the required tools
    75 if [ -x "/usr/bin/makedumpfile" ] && [ -x "/sbin/kexec" ]; then
    76     GRUB_CMDLINE_EXTRA="$GRUB_CMDLINE_EXTRA crashkernel=384M-2G:64M,2G-:128M"

    74 # add crashkernel option if we have the required tools
    75 if [ -x "/usr/bin/makedumpfile" ] && [ -x "/sbin/kexec" ]; then
    76     GRUB_CMDLINE_EXTRA="$GRUB_CMDLINE_EXTRA crashkernel=384M-700M:64M,700M-:128M"

Once you have made this change, be sure to update grub:

sudo update-grub

so that the chnage will take effect. You will also want to reboot. Then you can validate that change by inspecting the boot command line:

cat /proc/cmdline

and see that the new value is now shown.

ubuntu@bug1195328-1210:~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.5.0-36-generic root=UUID=39eb48d3-958a-48e0-896e-b6b03cc2342a ro crashkernel=128M console=tty1 console=ttyS0 earlyprintk=ttyS0 rootdelay=300

Reference Material

The official references for configuring Ubuntu for kdump are here:

and you should refer to them for procedures for testing and verifying your crashdump setup.

Micosoft Azure has some notes on the kernel modules here:

Sunday, July 14, 2013

Fun with QR Codes

In honor of xkcd's cartoon today, I thought I'd make myself a qr code--a self portrait.

Be sure to follow the QR code in xkcd a couple times and then come back and try this self-portrait. Oh, and you can do your own QR code art at

Tuesday, July 9, 2013

OpenStack Programs Core Developers

It seems to be something I look up fairly regularly: A listing of OpenStack core developers--either so I can get a +2 or just because I need to know if someone is on or not-on a given list.

I found the Canonical list by initially proposing in this blog the wrong list--and apparently that's a fairly common problem. So I'll link to the real list and explain how you might also be referring to the wrong list as well.

Official OpenStack Programs
OpenStack Technical Committee
Lastly, I ran across OpenStack Planet Core Developers when I was creating this list. They may be an aggregator team (ie, bloggers who get aggregated).  And hopefully, this blog will show up in the planet soon.

Now for the wrong list. Not so long ago, much of OpenStack was managed in Launchpad. Consequently, there are also a somewhat correlating list of -core projects in Launchpad. However, I'm not going to reproduce them herein in order to avoid perpetuating them. I will mention the bug that has been posted to help clean them up though: and it is listed as in progress and did see some activity last month. If you happen to stumble onto this blog post and have some ownership over those dangling -core teams or other defunct/obsolete launchpad teams, go ahead and clean them up (pretty please).

Oh and one other editorial footnote: OpenStack now refers to the individual areas of development as programs, not projects as it used to. You might want to update your mental model to that terminology. Many thanks to ttx for the review of this document (though all errors and faux pas are mine.)

Friday, July 5, 2013

VirtualBox Host-Only Networking

VirtualBox allows one to configure a VM with host-only networking. This can be useful if you are connnecting a number of VMs together and need to put them on the same switch/bridge.

However, it's darn frustrating to figure out how to enable it as all of the googling and manuals indicate you just enable it by selecting settings within the VM.

What they fail to mention though (but is covered in the built-in help in VirtualBox), is that you must first create a device for this host-only networking to use. From the main VirtualBox window, choose Preferences (below the File) menu. That will bring up the global preferences for the VirtualBox hypervisor. Click "Network" and then the "+" sign to add a host-only network device (typically vboxnet0).

It's possible that some versions of VirtualBox create one of these at install time, but on Ubuntu, such is not the case. Now you can create a number of virtual machines and put them on the same nic. Your host OS will also now show that same nic. Here's an example from my laptop's OS:

medberry@handsofblue:~$ ip a show dev vboxnet0
6: vboxnet0: mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 0a:00:27:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet brd scope global vboxnet0
    inet6 fe80::800:27ff:fe00:0/64 scope link 
       valid_lft forever preferred_lft forever

and that same vboxnet0 is now an option when you select host-only networking in the vm:

Wednesday, June 26, 2013

Talk Like a Spy Day

There really ought to be a "Talk Like A Spy Day", something of a corollary to talk like a pirate day (tlapd).  tlasd would involve things like jamesbondisms, smileyisms, coldwarisms, etc.

Wednesday, January 30, 2013


I just wonder how many folks will be, like me, googling "magnificient" today after reading xkcd. I certainly did. I'm still undecided if it is a word or just artistic license.