Cinder Multi-Backend GOTCHA

When making puppet changes to handle OpenStack Cinder with multi-backends, we created a really painful problem. The multi-backend basically moves our Cinder (with ceph backend) from basically a default name of our cluster (like cluster01) to a backend specific name (cluster0@ceph) so that we can add additional backends (like cephssd and solidfire and 3par).

Unfortunately, this had the really bad side-effect of dropping the "host" that was providing just "cluster01" service. All attached volumes continued to work fine. However, it became impossible to launch new instances with these volumes, terminate (properly) instances with these volumes, or delete these older volumes.

The fix (once you understand the problem) is very straightforward:
cinder-manage volume update-host --currenthost cluster01 --newhost cluster01@ceph

NOTE: Don't do this unless these are really the same services. Presumably safe if you have a single cinder backend and are prepping for multi-backend.

OpenStack Clients on Mac OS Yosemite

Clearly I haven't written a blog post recently enough. That must explain the negative karma behind running into a tricky problem.

As we all know, "pipin' ain't easy". I held off upgrading to Yosemite for a long time so as not to break my functioning Mac with OpenStack clients. However, I found time and reason to upgrade last weekend and had helped several others through the Mac client solutions a week prior.

Alas, it was not to be a sweet journey for me. I endedup with the dreaded xmlrpc_client issue. Some coworkers determined that you could resolve this by uninstalling "six" and then reinstalling "six". Unfortunately that doesn't really work. A "sudo -H pip uninstall six" did do the right thing, but "easy_install six" never succeeded. And I should note the reason that six in pip doesn't work in the first place is that the Mac OS Yosemite itself has a downrev version of six installed.

The trick, at least in my case, was to "type -a easy_install" and note that there are two versions of easy_intall. Pick the second one and you are off to the races. Here's the steps if you are still reading:

dmbp:~ dmedberry$ sudo easy_install six #fails
Password:
Traceback (most recent call last):
  File "/usr/bin/easy_install-2.7", line 7, in
    from pkg_resources import load_entry_point
  File "/Library/Python/2.7/site-packages/pkg_resources/__init__.py", line 3020, in
    working_set = WorkingSet._build_master()
  File "/Library/Python/2.7/site-packages/pkg_resources/__init__.py", line 616, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/Library/Python/2.7/site-packages/pkg_resources/__init__.py", line 629, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/Library/Python/2.7/site-packages/pkg_resources/__init__.py", line 807, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: setuptools==1.1.6
dmbp:~ dmedberry$ type -a easy_install
easy_install is /usr/bin/easy_install
easy_install is /usr/local/bin/easy_install
dmbp:~ dmedberry$ sudo /usr/local/bin/easy_install six