What did we have?
We use Chef for infrastructure management at Safari Books Online. We have a Production Chef (ChefProd) server that performs standard tasks such as putting users on machines and setting up monitoring. Chef cookbooks also perform specific tasks such as setting up web, database, and application servers. In addition to the Production Chef environment, we also have Chef Development (ChefDev) environment used for testing Chef cookbooks.
We run a Gitlab server for version control. Gitlab is the open source project for self hosted git management. It is a git server and web UI written in Ruby on Rails and it works great.
What did we want?
- Goal #1: No knifing in anything to production manually.
- This was for fat finger errors as well as a lack of version control backup.
- Goal #2: Everything in Chef was in version control.
- Goal #3: To have our entire Chef ecosystem within a testing framework.
How we did it.
We built a Jenkins VM (provisioning users and monitoring through Chef) for our CI (Continuous Integration) testing server. Jenkins is a great tool for testing out software.
Each Chef cookbook is stored in version control with Gitlab. For each git cookbook repository, we made an identical Jenkins job. This Jenkins job would run all the tests on the cookbook on one of its nodes, searching for bugs. We added a git hook on merge to master for each cookbook. This means every time one of us git committed and merged to master, Jenkins (who was always listening for the Gitlab hook) would fire off the cookbook test.
All this happened in the Chef Development (ChefDev) environment and we wanted to accomplish Goal #1, which was put tested code into production without using knife. We needed a way to push changes to production, the simpler, the better. We arrived at having a trigger file, i.e. a file that would send the signal that we wanted to push to production. If and only if this trigger file was git committed and merged to master, Jenkins would deploy to production. We decided to use a CHANGELOG file (with the added bonus of frictionless deploy comments) but any file would suffice.
With this system, we could test out cookbooks on the ChefDev environment, and have Jenkins knife them into production at will. All code was in version control and tested. This fulfilled Goal #1, Goal #2, and Goal #3.
Vagrant makes development better. We use the above workflow and leverage Vagrant VMs for local testing before this entire process. With new cookbook development, it’s nice to be able to move quickly into writing tests and software. This is where Vagrant shines! We made special Vagrant VMs that mimic our production environment. DevOps can test new cookbooks locally before creating Jenkins jobs and Gitlab hooks. Existing cookbooks can be tested locally before the Jenkins tests, so you can develop and iterate quickly when starting out on a new feature.