I know I’m late to this party, but I just have to say it: I absolutely love Ansible.

For many years, I have been a Puppet wrangler.  It was always a real struggle.  I mean, the basics are pretty straightforward, and the framework provides everything you reasonably need to achieve some exceptionally sophisticated automation.  If you embrace the “Puppet Way”, there’s really nothing you can’t do.  I’ve helped write and maintain modules that are many tens-of-thousands of lines of code that pretty well ran our application infrastructure and was the foundation of the application’s software delivery machinery.  But it was a pain.  It was never just right.  There were always caveats and mental hoops to jump through.  It made me feel inadequate as a developer.  It broke my brain.  It was such a chore.  I came to dread wading into module code for any reason. Continue reading

The Essence of Engineering and Personal Neuroses

I read an interesting post this evening by Michael DeHaan, titled The Archaeology of Software.  In it, he makes the case that there is a great deal of value in endeavoring to understand why things exist as they do in, specifically, the field of software development.  He says (I’m paraphrasing) that’s way more productive than dismissing something out of hand as being subpar, dumb, brain dead, or whatever – and a learning opportunity is missed.  He makes a lot of insightful points, and I highly recommend you spend a few minutes reading it.

I know he was making a larger point about how some people behave towards others’ solutions and ways we might strive to rise above pettiness and vitriol to be more constructive, but he touches on so many facets of engineering.  What struck me as I read his post was how much engineering wisdom there was implicit in his words.  I don’t know Michael, and can’t say how deliberate his wording was, but he at least casually touched on the essence of engineering a lot in the post.  He spent a great deal of time enumerating a long list of  heuristics he has developed over time (see my last post).  These heuristics are very obviously hard earned from experience, and are all very consistent with the essence of engineering.

Continue reading

The Process

In my last post, I spent a long time describing some recent successes at work.  At the end, I had the nerve to close the post on a very unsatisfying allusion to a ‘discussion for another post about “the process”‘.  Let this post be the first installment in a recurring series discussing “the process”, by which I mean “Engineering process”.

I want to start by addressing something that wasn’t acknowledged in my previous post, and is really only obvious in retrospect:  While the successes were pretty spectacular, they were made possible by a situation of extremely pent-up need.  That string of rapid, extreme successes was the dam breaking more than anything.  We were just there to ride the wave.  We made it look easy and sound amazing, but the truth is the situation made that easy for us.  Since that initial burst of productivity, things have settled down to a steady plod as we continue to iterate on improving things.

Continue reading

Unicorns and Rainbows

My employer has been in the process of reorganizing for several months, now. A couple of weeks ago, as part of this reorganization, I was moved to a “new” team. In actuality, this team is simply a small subset of the people I already worked with. It was an All-Star team of three. Our mission was … whatever we saw fit, related to everything in our domain of skill, interest, and concern. Mostly we would direct our efforts at release engineering, build engineering, systems architecture, and deep-investigation type troubleshooting. Stuff we already did anyway, albeit much less formally.

When I first learned of the new team (and the team reorg plan in general), I was in a bad state of burnout. So bad, in fact, I just couldn’t even get excited about the new team. I just didn’t care. However, when the three of us got into a room with our PO for the first time, the energy was palpable. We are like-minded engineering types, and I have the utmost admiration for my teammates’ skill, professionalism, and opinions. I felt much more positive after that first meeting.

Continue reading

Engineering Method

Lately, I’ve been studying Engineering Method by reading the textbook “Discussion of the Method“, by Billy Vaughn Koen.  It has been a very enriching exercise.  From reading this book, I have acquired a formal language to express many thoughts and observations I’ve had for years; but, it has also been great because I have learned new ideas and made connections I hadn’t noticed before.

My grand vision is to acquaint myself with the material in this book through a study group with one other person.  From this study group, we are developing a set of talking points with the intention to (a) begin spreading a cogent, coordinated message about engineering practices and (b) lead a small reading group of 5 or 6 people to train another generation of budding engineers within the organization who can go forth and reinforce the message.  Eventually, I’d like to condense the material down to a meaningful set of highlights we could use to develop a recurring, short training course on the topic; perhaps make it an annual session we subject everyone to.  You might recognize some of these patterns if you’ve ever read the book “Fearless Change“, by Mary Lynn Manns, Ph.D. and Linda Rising, Ph.D.

Continue reading

Reset: Quasi-Post-Mortem for the 2015 Plan

Life has unfolded in completely unexpected ways since my last blog post.  If I was grading myself on my progress of the goals I set forth in the post titled The 2015 Plan, I’d say:


C.  I have written tests for the projects I’ve undertaken and continued at work.  I’m actually off to a decent start on this particular goal.


F.  I’ve been bashing (and Python’ing) my way through everything.  Maybe someday.  At this point it has been nearly a year since I’ve written anything in Go and I’ve forgotten more than I remember.


F.  When I write something to put out there, I’ll put it on github.  The aspiration behind this goal was to practice coding more often and at higher intensity.

Grad School

F.  I looked into testing dates, found myself in limbo for various reasons, and never got around to scheduling a time to take the GRE.  Of course, I haven’t opened the application process anywhere, either.  Given some life changes on the near-term horizon, I’m glad I haven’t committed to a program just yet.


What Actually Happened Instead

Stuff.  You might’ve read my list of goals and thought “this guy is overly ambitious,” which is absolutely true.  But mostly, I just wasn’t observing any practical limits regarding focus.  I have been suffering  of lack of focus for quite a while.  I’m now on a journey to regain that state of focus I once possessed.  I’ve made an ordered list of what I think is important and found that some of the goals didn’t really match up with things on that list.  So, I’ve given myself permission to let those things go.  My approach to regaining focus is selecting the two most important things on my list and giving them all my extra available attention while still doing a passable job at everything else I have to do; like being a dad and husband and employee.

At any rate, I’m retiring the original 2015 Plan, and we’ll speak of it no more.

The 2015 Plan

Six days late, but these are a few of the things I plan to focus on in 2015:


I automate.  It’s what I do.  That means I write lots of code.  Sadly, I possess rather underdeveloped testing foo.  Granted, there aren’t elaborate frameworks for bash (What? You want me to write bash scripts to test my bash?); and our implementation of puppet is mind-meltingly sophisticated coupled with lack of (administratively authorized) tooling to programmatically spin up virtual machines, making it sufficiently non-trivial to automate testing of manifest changes (and other forms of automated testing for other scenarios …*SIGH*).   But, I could still exercise my testing foo more than I do.


I specifically said “Go”, not “programming”.  I program every day.  I am formally declaring right here, right now, though, that if bash isn’t the most appropriate tool for my programming task I will use Go, given a choice.  Our use of Go at work has exploded in the last 6 months.  I love that it’s fast like C yet accessible like python.  I also love that it’s statically compiled.  It’s so great for performant, self-contained, portable utilities and services.


I want to do a better job of putting my projects out there.   It’s a nice way to give back.  But, there’s a lot of really cool stuff that hooks into github these days that I want to play with.  So, I can get a twofer with this goal.

Grad School

I will be applying to grad school this coming application cycle.  Maybe this time next year, I’ll have a semester of grad school under my belt.

Bootstrapping A Simple Openstack Lab

In my last post, I waved my hands and assumed you had a deployment of Openstack to run the CoreOS cluster that was the subject of that post.  I felt a little absurd talking about autonomous, self-assembling infrastructure-in-a-box while choosing to forego a simple tutorial on setting up an all-in-one lab environment, at a minimum. So this time around, we’ll discuss a way you might bootstrap a simple Openstack deployment.  This setup will result in an RDO all-in-one Openstack system where the floating ip pool will be assigning valid IP addresses on your “public” network.  In this example, we pretend your lab subnet is  When you assign a floating IP address, any host that can route to that network will be able to talk to those floating IPs (assuming your security group allows it).

In my personal home lab, the first thing I set up was a Cobbler server.  I performed a minimal installation of CentOS 6.5 x86_64 with a netboot ISO, installed the EPEL repo, and set up Cobbler using the instructions on the Cobbler website.  I configured Cobbler to manage DHCP and DNS, and also installed cobbler-web.  I imported one distro: centos-6.5-x86_64.

Continue reading

A Simple CoreOS and Etcd Cluster

In my idle moments, I contemplate such things as self-assembling infrastructure-in-a-box.  What I mean by that is how do you go from an an empty room (conceptually) to doing useful work for the business in as few steps as possible, as quickly as possible?  What minimum complement of services do you need, and how do you bootstrap it into existence, autonomously?  The goal is to perform as few manual steps as possible to get to a state where you can easily build out the rest of your infrastructure and start deploying applications right away.  Bootstrapping.  Inception.  Whatever.

In this post, at a high level, I want to lay out a simple application platform bootstrapping exercise.  We’re going to make use of a HOT template for Openstack to bring up a three node CoreOS cluster which performs auto-discovery to assemble an etcd cluster.  This is not a tutorial on Openstack.  I am not covering the obvious question of how to get Openstack running.  The point of this exercise is only to lay the foundation for running clustered applications in Docker containers.

Continue reading

Continuous Agile

Even though my organization has been practicing “agile” for several years, I have witnessed a seismic shift in what that means over the last 15 months.  We have shifted from Sprint Teams running two-week Sprints and releasing every other Sprint cycle to an interrupt-driven Kanban “pull” model with no strictly time-boxed development cycle where we release twice a week, whatever is ready.  This is just a step along the road to fully automated Continuous Delivery for us, though.  We’re learning as we go.  Our focus right now is ruthlessly stripping our manual release process to its essence, then automating what’s left.

Even though we’re focused on optimization of a semi-manual release process, we’re thinking about what else we need to do to improve our game.  We’re not just thinking about release process.  We’re still defining our development philosophy is, how we can create an infrastructure in which we can experiment, and what techniques we can employ now that improve our flexibility and give us more options for future experimentation.

Continue reading