Juju environment in the HPCloud

I've been an HPCloud user a long time. They rocked the cloud world as one of the first OpenStack deployments open to the public.

And I work at Canonical, so Juju is key to my cloud usage. There are some "howtos" for getting the hpcloud to work with Juju but I was just missing a few things... not sure if it was out of date or what, but the following worked fine.

Create an "environment" within your ~/.juju/environments.yaml that looks like this:

default: hpcloud
environments:
  hpcloud:
# Get api-keys here:  

#  https://console.hpcloud.com/account/api_keys 
# after you have a valid hpcloud account and login
    access-key: **************:****************
# from the api-key page use the access key 
#     with a central colon (:)
    secret-key: *************************************
# secret-key is just below the access-key you used above, 
# typically hidden/obscured by default
    juju-origin: ppa

# This was the most important line.

# juju-origin defaults to "distro" and the ubuntu in hp cloud

# currently is precise without an upgraded juju, so without

# setting this to ppa, you have a catch-22
    control-bucket: make-your-own-bucketname-cb
#make this up
    admin-secret: make-up-your-own-admin-secret
#make this up
    region: az-1.region-a.geo-1
    default-image-id: 8419
# use an appropriate image number from the selected region
#  you are using in the next line
#  8419 is valid for az-1
    project-name: hpcloud@medberry.net
# This is shown as the tenant name.
#  yours may look more like email@example.net-tenant-name
    default-instance-type: standard.medium

# This value while required, is ignored since Juju added constraints

# you can set it to any arbitrary string

# use "juju set-constraint instance-type=standard.medium" 

# to actually set a default instance type after you bootstrap

# and you can set the default at bootstrap time

# with "juju bootstrap --contraint "instance-type=standard.medium"

    auth-url: https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0/
    auth-mode: keypair
    type: openstack
    default-series: precise

By default, your ~/.ssh/id_rsa.pub will be injected into the instances. You can pass an alternative in by adding an:

 authorized-keys: ssh-rsa AAAAPUBLIC_KEY_TEXT_HERE

in the hpcloud environment stanza.

A little issue with nova keystone and cacert.pem

I was having a bit of trouble with OpenStack as I was setting up a demo. The demo included the normal essex bits and pieces: nova, keystone, horizon, glance, rabbitmq, mysql, and swift. Most of the way through the setup, I came to this step:

cd /home/ubuntu/creds
. ./openrc
nova x509-get-root-cert

and got a curious error:

ERROR: string indices must be integers, not str

which is correct, but not all that meaningful. I'll jump to the solution now. Somehow I had glossed over the step to create the nova user in keystone. Once I did that the get-root-cert worked.

Various error messages about this (and askubuntu answers) abound and include helpful suggestions like check your endpoints. (My endpoints were all fine.) And other suggestions (though I've only seen it once--as I was writing this blog post after I had a solution) to "Check that your service users can authenticate against keystone."

That was the actual issue--my "nova" user was never created (as I've said). And although the openrc file I had created doesn't reference this user, it is used by keystone to get the information.

I was able to troubleshoot this (on a multihost system with nova controller on node B and keystone on node A) by simultaneously tail -f /var/log/**ALLTHENOVALOGS** and tail -f /var/log/keystone/keystone.log where I plainly found:

> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] ******************** REQUEST BODY ********************
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] {"auth": {"tenantName": "service", "passwordCredentials": {"username": "nova", "password": "novapassword"}}}
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi]
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Matched POST /tokens   
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Route path: '{path_info:.*}', defaults: {'controller': }
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Match dict: {'controller': , 'path_info': '/tokens'}
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Matched POST /tokens   
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Route path: '{path_info:.*}', defaults: {'controller': }
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Match dict: {'controller': , 'path_info': '/tokens'}
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Matched POST /tokens   
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Route path: '/tokens', defaults: {'action': u'authenticate', 'controller': }
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Match dict: {'action': u'authenticate', 'controller': }
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] arg_dict: {}
> 2012-09-24 19:26:47  WARNING [keystone.common.wsgi] Invalid user / password
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] ******************** RESPONSE HEADERS ********************
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] Content-Type = application/json
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] Vary = X-Auth-Token 
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] Content-Length = 89 
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi]
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] ******************** RESPONSE BODY ********************
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] {"error": {"message": "Invalid user / password", "code": 401, "title": "Not Authorized"}}
> 2012-09-24 19:26:47    DEBUG [eventlet.wsgi.server] 172.25.100.32 - - [24/Sep/2012 19:26:47] "POST /v2.0/tokens HTTP/1.1" 401 248 0.030605

Once the nova user was created properly, the cacert.pem is created.

Sadly, this user/password is created many many steps earlier (and typically on another node) during the keystone config and only tested near the end of an install.

Hangout How To

I found myself googling how to make Google Hangouts follow speaker. Sometimes I click on a specific speaker to put their webcam centermost in my display. However, switching back to "focus follows speaker" mode was not obvious. Simply click on the speaker (at the bottom) that you already have selected and a green outline around the speaker will turn off. This "toggles" the focus follows speaker mode (off or on). Clicking on a different speaker will just move "permanent" focus to the other speaker, so you must select the current speaker (which is more like a "de-select".)

YAHTNG

Yet Another Helping The Next Guy

I bought my son a shiny new Ivy Bridge based Asus laptop from dealnews for high school graduation. Of course, he only runs Ubuntu, but I wanted to keep the Win 7 64 around for flashing BIOS and related reasons (and someday he may need to run some Windows program....)

It's pretty straight-forward to install Ubuntu alongside windows in a dual-boot fashion as the Live installer knows how to do this. But apparently not on some UEFI based systems--or that's what I thought. It turns out, the installer knows how to work with UEFI systems as well. It just doesn't know how to do it on a system with a broken/corrupt/obscured GUID Partition Table (gpt).

I've used gpt for years as I used to work in HP's Itanium Linux lab. And I've seen some GPT tables corrupted before. The ASUS (at least this one) ships with the secondary GPT corrupted. The last partition (I think 5th) somehow extends INTO the secondary GPT. (Think of the secondary as a backup table in case the primary gets corrupted--a failsafe.) This corrupt secondary table makes the partition table unreadable or possibly unusable to Ubuntu (and moreover, to the underlying libparted and the parted utility.)  The installer just "determined" that the disk was unformatted so it kindly offered to just install Ubuntu to this blank disk (destroying all the existing partitions in the process.) Not so good.

I downloaded gdisk (from Universe) as it seems to be designed from the ground up for GPT tables and it was able to read the partition table and show me the partitions. These matched up with what /proc/partitions was showing me (which seemed to be right all along.) It also matched up with what I was seeing in Windows 7. ASUS ships with a C drive, a D drive, and a recovery drive along with the normal EFI system partition (ESP) and a Microsoft Reserved Partition (MSR). So the layout looked something like this:


Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048          411647   200.0 MiB   EF00  EFI system partition
   2          411648          673791   128.0 MiB   0C01  Microsoft reserved part
   3          673792       339511295   161.6 GiB   0700  Basic data partition
   4       339511296       925575167   279.5 GiB   0700  Basic data partition
   5       925575168       976773167   24.4 GiB    2700  Basic data partition

Where 3 is C (OS) drive, 4 is D (DATA) drive, and 5 is the ASUS Recovery partition. It is partition 5 that seemed to be the source of the error:


Warning! Secondary partition table overlaps the last partition by
33 blocks!
You will need to delete this partition or resize it in another utility.


To resolve this issue, I simply deleted partition 4 and recreated it from the same start point but smaller and ending sooner. Then I dd'd partition 5 to another disk, deleted it and recreated it (same size) but now it ended before the end of the disk. I then restored it via dd.  (The partition Warning! went away as soon as 5 was deleted and did not reappear when it was recreated at a new start sector.)

I failed to grab a snapshot of the final partition table entries but I basically shortened 4 by a couple GiB (and didn't try to "nail" it to exactly 33 blocks shorter.)

Once the GPT table had a functioning secondary GPT, the Ubuntu installer was able to find it without problems. I just used gdisk to make the D partition into a Linux format partition (8300). (There was nothing on the D drive.) The installer did create a GRUB entry for the Win 7 but that doesn't really work. However, using ESC, I can select the Windows partition from the UEFI boot menu and it dual boots just fine.  After the installation was done, I upgraded from the Precise install release bits to the latest bits (including newer kernel) and the grub update worked fine as well--so it seems to be a rock solid dual boot machine now.

Holler if this helps you or if you know more about why ASUS formats the disk this way--there could be an underlying reasons. Oh, and one more thing, I actually did make the Win 7 recovery disks PRIOR to doing any Ubuntu at all and I've not tried to do anything since with the recovery partition.

You can reach me via the comments below or email me at asus DOT dowdberry AT medberry DOT net.

And some final words: Ubuntu Live Installer makes an awesome detective kit and this Ivy Bridge notebook truly rocks.

Helping the next guy

Just a bit of IRC discussion I thought I'd share. A friend was looking for a way to disable X on an Ubuntu install converted to be a mythtv backend. After a web discussion, another friend suggested passing "text" in on the cmdline. That's a darn good suggestion.

This works in both gdm and lightdm as they both search for this in their startup script (/etc/init.d/gdm and /etc/init/lightdm). Lightdm is the default display manager in Ubuntu 11.10 Oneiric and gdm was the default display manager in prior versions.