Posted on by & filed under infrastructure, IT, testing.

Here at Safari Books Online, we are building an increasingly robust infrastructure, and we have made automation a top priority so that we spend our productive time working on productive projects instead of fighting fires.  As we grow to support new and different types of web-based apps and clusters, not to mention desktop workstations and development VMs, it’s been important to manage our infrastructure using configuration management software.  For this job, we’ve chosen Chef open source server to keep our systems tuned. We’ve taken special interest in building tools to provide an operational awareness that will enable us to quickly and easily ensure that services are behaving as expected.  Over in the IT operations team, we had been previously sold on the benefits of Behavior- and Test-Driven Development by our friends in Engineering, and in recent months, we have begun to include mechanisms for testing our code changes in the configuration management software we use throughout our infrastructure.

We operate with the general idea that if you don’t test for a condition and you don’t have an alert for it, you can’t reasonably assume you know it’s working properly.  Moreover, in the spirit of just doing the Right Thing just once (in other words, DRY), another benefit of infrastructure software testing is that after you’ve defined your needs and written your tests, you have a a pretty good idea of what kinds of alerts you need to create in your monitoring software.  For Chef, developers have created tools for both unit tests and integration tests. We looked at a number of them, scouring the Food Fight Show podcast archives for any references to testing and using narrow search terms to dredge up various Opscode mailing list threads about testing from the past year or so.

At first, we experimented with chefspec and, in fact, wanted to use it and write primarily unit tests for local testing because it seemed like the most lightweight manner in which to test whether a cookbook behaves as expected.  It was simple to get up and running, but we quickly encountered pain when it came to stubs and mocks.  Apparently, like the other testing tools, chefspec is still under active development and may eventually find its way into our toolbox, but around the time we were having difficulty integrating it into our workflow, one of our developers had the opportunity to attend DevOps Days in New York and spoke with Sean OMeara from Opscode who demonstrated minitest-chef-handler, which painlessly facilitates integration tests.  In fact, the adoption of minitest-chef-handler led us to a broader revelation about how we should expect to use Chef.

As we experimented with the different testing and workflow tools, we were, maybe without even realizing it, looking for The Right Way.  Indeed, our bias towards chefspec at first was rooted in the idea that we would follow BDD/TDD philosophy closely and write the tests before the cookbooks, which is theoretically easier with unit tests, which are roughly simulated chef runs and typically more abstract than the integration tests.  Upon coming to know and love minitest-chef-handler so easily, along with the minitest-handler cookbook, it became apparent to us that to get the end results we want — in early 2013, at least — is to accept that Chef and its periphery are in a period of rapid development, particularly when it comes to issues related to workflow and testing tools.  As a result of this, we decided to stop worrying and learn to love transient infrastructure tooling.

Currently, we have a workflow in which the master of each cookbook’s repo can always be deployed, and we run integration tests on a staging node before pushing out to prod.  We have a few safeguards in place to prevent bad cookbooks from making it out to production, but we certainly don’t have fully automated infrastructure at this point.  We do, however, have a workflow in place that supports multiple developers and gives them somewhere to test their cookbooks other than production, and they can do so without undue psychological overhead.  Moreover, to get to where we are now, we had to get detailed about how we want to interact with Chef and what we want it to do, which necessarily led us to determine how we want to comfortably build out Chef while guaranteeing our infrastructure.

In case it seems coy to obliquely describe our environment and development pipeline without details, it’s specifically to avoid muddying the somewhat confusing landscape of Chef testing software.  Indeed, “Chef” as a metaphor holds up well because we have learned so far that it helps to know how to cook with the various ingredients in order to determine which you want in your entrees.  While we were seeking tools, we doubtlessly benefited from many tools discussions within the Chef user community, but even today, we can say that we already have our eyes towards the adoption of other testing tools in the near-ish future (Test Kitchen is coming together and integrates with Berkshelf).  We wanted to share our story, though, so that we might provide hope to other Chefs by saying that we have implemented an imperfect system and are embracing this experience as real-world iterative development.

Tags: chef, configuration management, devops, PaaS, TDD,

3 Responses to “Building and Testing Infrastructure with Chef”

  1. Martin Cozzi (@martincozzi)

    Thanks a lot for taking the time to write about your workflow. I agree with you that Chef is in a period of rapid development and that it needs to be embraced as is.

    My question for you is: Why did you chose one or the other, instead of having both work together?
    I feel like ChefSpec does a great job at TDD/unit test, while minitest-chef-handler does a better job at integration tests.

    For example, if you write a cookbook that needs postgresql installed, ChefSpec could take care of the basic stuff like verifying the requirements, or validating your Resources (file exists etc.) while minitest-chef-handler could take care of verifying that the process has actually started.

    Like you said things are moving fast, and I am experimenting as well. Keep up the posts! Looking forward to see where you guys are headed.

    • JT Gray

      Martin, thank you for your comments and questions.

      As a team of back-end developers responsible for deploying stable infrastructure that gets a decent amount of attention post-deployment, our goal in testing was initially to implement reliable acceptance testing. Your point about unit vs integration testing is well-made, but in our case, we prioritized choosing an easily adopted solution that we could use for testing changes in a production-like environment at a pretty brisk pace. At the time of our initial test development, it was pretty hairy trying to stub searches and attributes, while running minitest tests against a dev chef-server was simple to accomplish.

      But that was v1; check back soon to hear more about the next iteration!


  1.  Using test-kitchen with Berkshelf, LXC and chef-zero | Safari Flow Blog