Monday, July 27, 2015

Time Warner Cable (TWC) OpenStack Summit Talks in Tokyo -- Vote Now

(Using my soapbox to pitch some Time Warner Cable proposed talks for Tokyo.)

Many of these are our experience running an OpenStack Cloud in production at Time Warner Cable. Others are particular areas of interest. And one is just for fun....

Click the shiny link to vote up or vote down, however you feel.

Abstract Author
More Info... Voting (link to abstract on
Duct Tape, Bubble Gum, and Bailing WireEric Peterson, TWC,
GoDaddy and others
Fernet Tokens: Why They're Better and How to SwitchMatt Fischer and the Frenet teamOps
Moving a Running OpenStack CloudMatt Fischer
Eric Peterson
Customer Pain Points Revealed in TWC's OpenStackKevin Kirkpatrick
David Medberry
Building the right underlay, lessons learnedSean Lynn
Darren Kara
Monitoring OpenStack with Monasca -- Totally Worth the EffortBrad Klein
Ryan Bak
Overcoming the Challenges. LBaaS in practice at TWCJason Rouault
Sean Lynn
Upgrading OpenStack Without Breaking Everything (Including Neutron!)Clayton O'Neill
Sean Lynn
Integration & Deployment Testing of an OpenStack CloudClayton O'Neill
Matt Fischer

OpenStack TriviaDavid Medberry
Kevin Kirkpatrick

Owls, Cats, Toads, Rats: Managing Magical Pets - VM persistenceDavid Medberry
Craig Delatte
Enterprise IT Strategies
Other Ways to ContributeDavid Medberry
Eric Peterson
How To Contribute
LibVirt: Where are we today?David Medberry
Sean Lynn
Related OSS Projects
OpenVSwitch: Where are we today?David Medberry
Sean Lynn
Related OSS Projects

Monitoring: How to get the outcomes you want and want the outcomes you get!Steve Travis
Ryan Bak
Brad Klein
Monitoring / Operations
An all SSD Ceph cluster: Tips, Tricks, and lessonsBryan Stillwell
Craig Delatte
The evolution of Openstack Storage Services at TWCAdam Vinsh
Craig Delatte
Enterprise IT strategies
Building a self-repairing cloud with AIRyan BakMonitoring / Operations
Configuring Swift Using Puppet-swift: considerations and examples.
Adam VinshOPS

Thursday, May 28, 2015

Request for Reviews from Online Stores

Dear googlemonopriceamazon, please don't ask me to do reviews for items delivered via slow delivery. If I request slow delivery, there is a REALLY GOOD CHANCE I'm not going to use the item right away. So, don't ask me to review it when I haven't even taken it out of the package. You should only ask for reviews when things are shipped overnight. (Though arguably, if you are in that kind of hurry you don't have time to do reviews.) So I guess, I'm just saying, stop asking for reviews altogether.

And if I could have expressed this in a brief way, it would have been a tweet.

And, no, don't file a patent for this. I freely grant this info to the world....

Sunday, May 17, 2015

OpenStack, a really big deal

Okay, most folks know I've spent the last 4 years involved with OpenStack and have been attending summits since Boston. This is the first time however I've been overwhelmed with the number of other folks attending the summit and we aren't even there yet.

I'm at 36,000 feet over NW USA headed to Vancouver from Denver on United UA 323. By sight alone, I've identified 10 different companies sending folks to OpenStack (and I'm sure there are many more companies represented on the flight that I don't know.) About 30 folks that I know (or at least know of) are on this very flight for a week of design, operations notes exchange, and marketing. WOW. I'm expecting more than 6,000 folks in YVR this week--maybe 7-8K.

Friday, March 27, 2015

Cinder Multi-Backend GOTCHA

When making puppet changes to handle OpenStack Cinder with multi-backends, we created a really painful problem. The multi-backend basically moves our Cinder (with ceph backend) from basically a default name of our cluster (like cluster01) to a backend specific name (cluster0@ceph) so that we can add additional backends (like cephssd and solidfire and 3par).

Unfortunately, this had the really bad side-effect of dropping the "host" that was providing just "cluster01" service. All attached volumes continued to work fine. However, it became impossible to launch new instances with these volumes, terminate (properly) instances with these volumes, or delete these older volumes.

The fix (once you understand the problem) is very straightforward:
cinder-manage volume update-host --currenthost cluster01 --newhost cluster01@ceph

NOTE: Don't do this unless these are really the same services. Presumably safe if you have a single cinder backend and are prepping for multi-backend.

Wednesday, March 11, 2015

OpenStack Clients on Mac OS Yosemite

Clearly I haven't written a blog post recently enough. That must explain the negative karma behind running into a tricky problem.

As we all know, "pipin' ain't easy". I held off upgrading to Yosemite for a long time so as not to break my functioning Mac with OpenStack clients. However, I found time and reason to upgrade last weekend and had helped several others through the Mac client solutions a week prior.

Alas, it was not to be a sweet journey for me. I endedup with the dreaded xmlrpc_client issue. Some coworkers determined that you could resolve this by uninstalling "six" and then reinstalling "six". Unfortunately that doesn't really work. A "sudo -H pip uninstall six" did do the right thing, but "easy_install six" never succeeded. And I should note the reason that six in pip doesn't work in the first place is that the Mac OS Yosemite itself has a downrev version of six installed.

The trick, at least in my case, was to "type -a easy_install" and note that there are two versions of easy_intall. Pick the second one and you are off to the races. Here's the steps if you are still reading:

dmbp:~ dmedberry$ sudo easy_install six #fails
Traceback (most recent call last):
  File "/usr/bin/easy_install-2.7", line 7, in
    from pkg_resources import load_entry_point
  File "/Library/Python/2.7/site-packages/pkg_resources/", line 3020, in
    working_set = WorkingSet._build_master()
  File "/Library/Python/2.7/site-packages/pkg_resources/", line 616, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/Library/Python/2.7/site-packages/pkg_resources/", line 629, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/Library/Python/2.7/site-packages/pkg_resources/", line 807, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: setuptools==1.1.6
dmbp:~ dmedberry$ type -a easy_install
easy_install is /usr/bin/easy_install
easy_install is /usr/local/bin/easy_install
dmbp:~ dmedberry$ sudo /usr/local/bin/easy_install six