Posted on by & filed under Devops, infrastructure, IT, Operations, programming.

What is Chef Provisioning?

Chef Provisioning is a drop-in library for Chef that gives developers and infrastructure teams an added dimension of automated system configuration: the ability to bootstrap and install a series of OS and configuration deployments onto “bare metal”. There are a variety of drivers that can be used as bare metal abstractions, including Docker, LXC, Fog (EC2 / DigitalOcean / OpenStack), AWS, Azure, Vagrant, VSphere, DigitalOcean, Hanlon, OpenCrowbar, and SSH. Chef Provisioning, combined with your choice of these drivers, provides a number of new abilities:

This elevates Chef’s potential role beyond its historic use for managing resources on individual servers. Chef is a proven platform to manage components, such as installed packages, sensitive data (for example, SSL keys, encrypted passwords, API keys), configuration templates, user accounts, and other aspects of server management. With Chef Provisioning, it can now define and manage the servers themselves.

How can I use it?

Chef Provisioning can be useful for local, decentralized development of:

  • Application software
  • Chef cookbooks
  • Docker repositories
  • System configuration (user authentication, alerts, logs, stats, NTP, etc.)
  • Load balancer health checks and settings

Chef Provisioning promises that you can take the same provisioning script you use for your local Docker deployment (see deploy-environment.rb below) and run it in one of the other platforms listed above. You can also mix and match platforms to run machines in different mediums by using multiple drivers. The deploy-environment.rb on your machine can be shared with others, to provide them a copy of your full software stack configuration — leaving out the site-specific configuration (such as passwords), if you desire.

Show me an example

Using Docker with Chef Provisioning is the quickest way to get up and running. Here’s an example using the Docker driver that creates a network of nodes on your local workstation. There are 3 types of nodes, spanning 4 instances: 1 load balancer, 2 application servers, and 1 or 2 database servers, depending on the environment. While the example here won’t do much beyond install packages and set Chef node data, it should be enough to get you started.

Goals:

  • Define all machines in the environment, including their Chef node data and recipe run_list.
  • Be able to add, query, update, and delete these machines via the environment configuration file.
  • Enable sharing of the environment configuration file, so others can run their own copy of the environment (with or without site-specific information such as passwords).
  • Version-control the complete environment configuration.

Steps:

1. Copy the below example to a file on your local machine: ~/deploy-environment.rb:

2. Download and install Docker

3. Download and install ChefDK

4. Install the chef-provisioning-docker library for Chef

5.  Set up your local machine to serve cookbooks via Chef Zero

6. Choose Docker as your driver, and go!

When all is done, you should be able to run docker images and see output similar to the following:

It’s like Chef is completing Docker, and Docker is completing Chef.

Note: One nice side-effect of Docker is data de-duplication. So while the “VIRTUAL SIZE” of the images totals 1912 MB, the actual size on disk is only 1135 MB.

Upon launch, these images will talk to a defined Chef server, which can be either your workstation’s local Development Chef Zero server or a production Chef server. For this example, we are using our local machine as the Chef server, so we need to launch it before the docker instances can check in.

Now you can launch the docker images by running one or all of the following:

This is a powerful pattern for application development. Sharing a single environment config file between your colleagues enables everyone to run everything on your local machine. Testing cookbook interoperability can be done quickly in local environments to spare resources on central hardware. Application and Chef Cookbook administrators may find this is a quick way to upload code and have it deployed in one or more fully networked environments.

Troubleshooting

1. The first time I ran through the above script, I got an error:

The problem was that the boot2docker VM running on my machine could not resolve my host machine’s hostname ( hostname -f). I added an entry in the VM’s /etc/hosts file, and the issue went away.

2. If you receive an error like  Chef::Exceptions::ContentLengthMismatch: Response body length 8006 does not match HTTP Content-Length header 15655., this Pull-Request may fix the problem.

3. If you make changes and they aren’t picked up, you can delete the node data from your Chef Zero cache by running  rm -fr ~/.chef/local-mode-cache/ or delete the Chef Zero server data by running  rm -fr ~/chef/nodes/ (a cleaner way to do this would be to knife node delete nodename -z ).

Cleaning up

To delete all of your local containers, run  docker images -q | xargs docker rmi # use rmi -f to force.

If it’s running, shut down Chef Zero via Control-C or  pkill chef-zero.

Next Steps

If you are interested in staying up to date with the latest developments in Docker, I recommend browsing through the upcoming Docker Cookbook. If you aren’t very comfortable with Chef, you can get a great introduction by reading Learning Chef. These are just two excellent resources out of thousands that are available at Safari Books Online. Please let me know in the comments if you have trouble with the example or have other feedback.

Tags:

3 Responses to “Building Docker Images with Chef Provisioning”

    • Shane Ramey

      Thanks Robert! What other topics would you like to see me cover?

  1. Chris

    i tried your example but when the chef-client running in the container tries to retrieve data from the chef-zero, nothing is coming back. It tries http://localhost:8889/nodes/ but gets a 404.

    I’ve set the chef_repo_path and cookbook_path in knife.rb. I run the chef-zero in the chef repo directory, and in here the nodes directory is populated with json files.

    So I can only assume the chef-zero is looking somewhere else for the node data. Any ideas what I might be doing wrong?

    Thanks
    Chris