Customizing Chef

Preface

Picture the scene. It’s 3 AM. An ops engineer or developer has just been paged. Her brain feels sluggish; simple tasks seem to take just that little bit longer and her body is telling her it’s still asleep. This state of tiredness is something that many of us in on-call rotations often deal with as part of our jobs, albeit not a particularly welcome one.

When debugging complex issues in this state, I find myself most grateful for the modern incarnations of configuration management (CM) systems that we as operations professionals are able to make use of today. Although configuration management has been around for a long time, it’s only in the last few years that CM systems like Chef, Puppet, CFEngine, Ansible, and SaltStack have come along to evolve these tools from loosely-coupled collections of shell scripts and wiki pages into the rich orchestration frameworks we have today.

When I get paged at 3 AM, I’m quickly able to introspect into any changes that might have been made to the server or system in question and kick-start the late-night troubleshooting process. Imagine if instead of being able to run one command to look at the running config on one of my servers, I instead had to trawl through an internal wiki to find the install document that detailed that specific configuration. This very process might in fact be what some of you are using right now, and to you I say: be strong, brave engineer! There is hope and its name is configuration management!

In this book, we focus on Chef, developed and maintained by Chef, Inc. Chef, Inc. describes Chef as a server and infrastructure automation tool, but it could equally be described as a CM system or using any of a number of other definitions.

Chef is an incredibly feature-rich and powerful tool designed to provide a framework for users of any of the major operating systems to automate anything they want. Simply put, Chef is a generic platform that provides a number of built-in tools, resources, and services to facilitate this automation. Typically, the people at Chef, Inc. do not advocate any of these as “the one true way.” And how could they? You, dear reader, are one of the greatest experts in the world on your particular infrastructure, if not the expert. You understand in detail the servers and operating systems you run; the software stacks that sit on top; and the unique structure, workflows, and processes that keep your business running. The chances are good that out of the box, Chef will allow you to automate a great deal of these systems, especially if you leverage the Chef Supermarket, where Chef users upload cookbooks and recipes for their particular systems and software.

But what do you do when you have a specific requirement for something Chef doesn’t provide out of the box? You customize, of course! One of Chef’s greatest strengths is that it is nearly infinitely customizable. You can hook Chef into your monitoring systems; you can modify the built-in tooling and processes; you can even add your own tooling. But where do you start? You may have a rough idea of what you’d like to achieve, but where in Chef should you add your custom piece? How do you make sure it will work nicely alongside your production environment? That’s where this book comes in.

I’m going to take you on a journey through the internals of Chef, exploring how everything fits together, the various places you can hook in custom code, and when it’s appropriate to use each one. We’ll also look at whether you should customize—although this book is very much about helping you to customize Chef, it would be a fallacy for me to tell you that just because you can customize something, you automatically should. As someone once said, with great power comes great responsibility.^[1]

Who This Book Is For

This book is designed for Chef users who are already comfortable with the basics of using Chef and writing simple recipes, and are looking to level up their Chef skills to better understand how Chef works and customize it beyond the features it offers out of the box. You might only be interested in customizing specific areas of Chef, such as creating your own resources and providers to use in recipes, or you may be looking for a more comprehensive understanding of how Chef functions under the hood. Either way, if you’re interested in customizing Chef, this is the book for you.

Going into this book, you should already be familiar with writing Chef cookbooks using the built-in resource types provided with Chef, and be comfortable using tools such as chef-client to carry out a Chef run on a node. You should have a good grasp of at least the Ruby needed to write cookbooks—you don’t need to know any object-oriented programming or any of Ruby’s more advanced features, but you should be comfortable with some of the basic techniques, such as variables and if statements.

If you’re not quite at that stage yet, I recommend you take a look at Learning Chef by Mischa Taylor and Seth Vargo, which covers in detail all of the Chef essentials required to follow the material presented here. By the end of Customizing Chef, my hope is that you will have a solid understanding of how Chef works under the hood, when and where it can be customized, and have the skills needed to dive in and start creating customizations of your own.

How This Book Is Organized

This book is broken down into several parts, each covering different aspects of customizing Chef:

Part I

The first part of the book deals with what you might consider the foundational material needed to customize Chef. In Chapter 1 we take a look at the history and design principles of Chef, moving on to examine the different Chef setups typically used and the prerequisites for the material in this book.

Chapter 2 introduces some Ruby concepts you need to know in order to progress from writing recipe code to writing customizations. Rather than trying to teach the entirety of the Ruby programming language, this chapter focuses on those aspects that will be used throughout the rest of the book.

We close off Part I in Chapter 3 by diving into the internals of Chef to look at how the different components of Chef fit together, the anatomy of a Chef run, and how to make use of the Chef source code to level up your knowledge.

Even if you’re only interested in customizing specific aspects of Chef, I recommend reading Part I in its entirety, as the material covered is universally applicable to all types of Chef customizations covered throughout the remainder of the book.

Part II

The second part of the book covers the various customization types Chef supports which allow us to modify and tap into the Chef run process. We start off in Chapter 4 by learning about how we can extend Chef’s Ohai tool to enhance the information Chef stores about our nodes.

Chapter 5 looks at the handler model used by Chef, which allows us to add customizable responses to various conditions that occur during the course of our Chef runs (such as the run starting, succeeding, or failing).

We finish off Part II in Chapter 6 by examining the rich and customizable event stream built up by Chef throughout the course of a run, and how we can tap into this event stream to modify the output produced by chef-client and leverage this event data in our own reporting systems.

Part III

The third part of the book focuses on the different sorts of recipe customizations Chef supports to allow us to level up our recipes with custom resource blocks of our own.

We start off in Chapter 7 by learning about definitions, the simplest type of recipe customization, and libraries, which allow us to incorporate custom Ruby helpers to power our recipe code.

Chapter 8 goes on to look at Chef’s lightweight resources and providers, which allow us to create more advanced and fully featured resources to use in our recipe code through the use of special domain-specific languages (DSLs), which abstract away much of the more complex Ruby code.

We finish Part III in Chapter 9 by diving into heavyweight resources and providers, which allow us to use native Ruby to create even more powerful resources and providers at the expense of some of the abstractions and simplicity afforded by their lightweight equivalents.

Because many of the customization types in Part III could be used to implement any specific recipe customization, this part of the book also focuses heavily on which customization types might be best to use in different situations.

Part IV

We close off the book in Part IV by looking at some of the other customization types supported by Chef, and some advice and resources for contributing customizations back to the community.

We start off in Chapter 10 by learning about the knife command-line tool supplied with Chef and how we can create our own plugins to implement custom functionality.

In Chapter 11 we dive into the Chef API to learn how we can tap into the data from our own scripts and tools that the Chef server stores, regardless of the programming language we choose to use.

We close off Part IV (and the book) in Chapter 12 by examining some best practices and advice for contributing customizations back to the Chef community, and some of the tools and resources that are typically used for distributing customizations.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames and file extensions, and directory and pathnames.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, cookbooks, data types, environment variables, statements, and keywords. Also used for commands and their output.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

Tip

This element signifies a tip or suggestion.

Note

This element signifies a general note.

Warning

This element indicates a warning or caution.

Using Code Examples

Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/jonlives/customizing_chef.

This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Customizing Chef by Jon Cowie (O’Reilly). Copyright 2014 Jonathan Cowie, 978-1-491-94935-1.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [e-mail].

Safari® Books Online

Note

Safari Books Online is an on-demand digital library that delivers expert content in both book and video form from the world’s leading authors in technology and business.

Technology professionals, software developers, web designers, and business and creative professionals use Safari Books Online as their primary resource for research, problem solving, learning, and certification training.

Safari Books Online offers a range of plans and pricing for enterprise, government, education, and individuals.

Members have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technology, and hundreds more. For more information about Safari Books Online, please visit us online.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

O’Reilly Media, Inc.

1005 Gravenstein Highway North

Sebastopol, CA 95472

800-998-9938 (in the United States or Canada)

707-829-0515 (international or local)

707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://bit.ly/customizing-chef.

To comment or ask technical questions about this book, send email to [e-mail].

For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments

Customizing Chef would not have been possible without the help and guidance of many friends, colleagues, and family members.

I’d like to thank the entire team at O’Reilly—Courtney Nash for helping me get this project off the ground, my editor Brian Anderson for his editorial nous and patience, my production editor Nicole Shelby for taking charge of getting the manuscript from first draft to print-ready, and everybody else involved in making this book happen. I’d also like to give special thanks to Mike Rembetsy and John Allspaw at Etsy for their encouragement, support, and tolerance while I spent nine months writing this book.

Many thanks also go to my tech reviewers: Seth Vargo at Chef Inc.; Gergely Hodicska at Ustream; Pete Bellisano, Ben Burry, Laurie Denness, Ryan Frantz, and Daniel Schauenberg at Etsy; and last but not least the Sparkly Devops Princesses Jennifer Davis at Chef, Inc. and Katherine Daniels at GameChanger Media. Thanks also go to Serdar Sutay and Daniel DeLeo at Chef, Inc., and Fletcher Nichol at Heavy Water Operations for their accessibility and sanity checking at various points during this project.

Most importantly, this book would never have been possible without the continual love, support, and infinite patience of my wonderful wife Agne throughout the entire project.

---

^[1] Though often attributed to Peter Parker’s Uncle Ben, the first literary usage of this phrase appears to have been by François-Marie Arouet, a.k.a. Voltaire.