A little issue with nova keystone and cacert.pem

I was having a bit of trouble with OpenStack as I was setting up a demo. The demo included the normal essex bits and pieces: nova, keystone, horizon, glance, rabbitmq, mysql, and swift. Most of the way through the setup, I came to this step:

cd /home/ubuntu/creds
. ./openrc
nova x509-get-root-cert

and got a curious error:

ERROR: string indices must be integers, not str

which is correct, but not all that meaningful. I'll jump to the solution now. Somehow I had glossed over the step to create the nova user in keystone. Once I did that the get-root-cert worked.

Various error messages about this (and askubuntu answers) abound and include helpful suggestions like check your endpoints. (My endpoints were all fine.) And other suggestions (though I've only seen it once--as I was writing this blog post after I had a solution) to "Check that your service users can authenticate against keystone."

That was the actual issue--my "nova" user was never created (as I've said). And although the openrc file I had created doesn't reference this user, it is used by keystone to get the information.

I was able to troubleshoot this (on a multihost system with nova controller on node B and keystone on node A) by simultaneously tail -f /var/log/**ALLTHENOVALOGS** and tail -f /var/log/keystone/keystone.log where I plainly found:

> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] ******************** REQUEST BODY ********************
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] {"auth": {"tenantName": "service", "passwordCredentials": {"username": "nova", "password": "novapassword"}}}
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi]
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Matched POST /tokens   
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Route path: '{path_info:.*}', defaults: {'controller': }
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Match dict: {'controller': , 'path_info': '/tokens'}
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Matched POST /tokens   
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Route path: '{path_info:.*}', defaults: {'controller': }
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Match dict: {'controller': , 'path_info': '/tokens'}
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Matched POST /tokens   
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Route path: '/tokens', defaults: {'action': u'authenticate', 'controller': }
> 2012-09-24 19:26:47    DEBUG [routes.middleware] Match dict: {'action': u'authenticate', 'controller': }
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] arg_dict: {}
> 2012-09-24 19:26:47  WARNING [keystone.common.wsgi] Invalid user / password
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] ******************** RESPONSE HEADERS ********************
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] Content-Type = application/json
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] Vary = X-Auth-Token 
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] Content-Length = 89 
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi]
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] ******************** RESPONSE BODY ********************
> 2012-09-24 19:26:47    DEBUG [keystone.common.wsgi] {"error": {"message": "Invalid user / password", "code": 401, "title": "Not Authorized"}}
> 2012-09-24 19:26:47    DEBUG [eventlet.wsgi.server] 172.25.100.32 - - [24/Sep/2012 19:26:47] "POST /v2.0/tokens HTTP/1.1" 401 248 0.030605

Once the nova user was created properly, the cacert.pem is created.

Sadly, this user/password is created many many steps earlier (and typically on another node) during the keystone config and only tested near the end of an install.