Learning CFEngine 3

Preface

This is a book about system administration. As any system administrator knows, there is no professional joy greater than seeing systems work consistently and perform their tasks flawlessly. And the joy is even greater if the systems need as little human attention as possible. Automating system administration tasks is not only a source of pride, but also an urgent need once the number of machines under our control grows beyond a very small number, as it is otherwise impossible to keep track of everything by hand. The number and complexity of computer systems have grown exponentially over the years, to the point where managing them by hand has become impossible for any single person. To this effect, CFEngine can help. CFEngine is a useful automation tool, but it goes well beyond that. It provides you with a framework to manage and implement IT infrastructure in a sustainable, scalable, and efficient manner. It allows you to elevate your thinking about systems so that you can focus on the higher-level issues of design, implementation, and maintenance, while having the certainty that lower-level details are handled for you automatically.

My road to writing this book started over 20 years ago, when I first became a Unix sysadmin at my university, working back then on a DECstation 5000 running Ultrix, a few SGI machines with Irix, and a Cray Y-MP/400 supercomputer with UNICOS. Even in that relatively simple environment, the challenges of doing everything by hand quickly became apparent. Over the years I have appreciated more and more the advantages of automating as much as possible all system management tasks. I first heard of CFEngine (still in version 1 back then) during my early years as a sysadmin, and over the years I loosely followed its development. Then in 2009 I got to work with CFEngine 3, and was immediately impressed with its flexibility and power. I also realized that a book about it was needed to help beginners overcome many of the questions that surface while learning to use it. Much of the literature at the time was focused on CFEngine 2, and CFEngine 3 is a completely new version, with vast improvements in all its aspects, including a completely new syntax.

It is a pleasure to finally deliver this book to you, and I hope you enjoy it.

Who Is This Book For?

This book is for you if you are a system administrator who is interested in learning new tools and techniques for making your life easier. I assume throughout the book that you are relatively well versed in system administration techniques, mostly about Unix-style operating systems. It will also help in some parts if you are familiar with regular expressions. I do not assume you know anything about CFEngine, but if you already know it, I am sure you will still find some interesting tidbits and learn some new techniques.

This book is not a complete reference to CFEngine. It is a “learning” book. The CFEngine manuals are an excellent source of reference information, and the text contains numerous references (mostly in the form of links embedded in the electronic versions of this book) to the appropriate documentation.

Overview of the Book

This book is organized as a progressive tutorial and is meant to be read from start to finish. If you already know some of the concepts you may be able to skip some of the basic sections. However, keep in mind that there are many examples and concepts that are developed over the course of a whole chapter (this is particularly true for Chapter 4), so you may be missing some of the context if you skip ahead.

On the other hand, I have read enough books myself to know that most people are unlikely to read it from start to finish. So most sections are as self-contained as possible without being repetitive, and with ample references to other sections when necessary. This book consists of seven chapters:

Chapter 1 is for motivation and historical perspective. It describes the many benefits that can be obtained through pervasive system automation, and describes the history and versions of CFEngine.

Chapter 2 is for quick and easy practice. In it I will walk you through getting CFEngine up and running on your system, and then writing and executing your first CFEngine policy.

Chapter 3 gives you a needed conceptual foundation. In it you will still see plenty of examples and CFEngine code, but with an eye on teaching you the basic principles of how CFEngine works, both from a theoretical (e.g., promise theory) and practical (e.g., language structure and features) point of view. You will also find pointers to many useful sources of information about CFEngine. You will probably refer back to this chapter often as you read through the rest of the book.

Chapter 4 is for really diving in. In it we will go through many examples of different tasks you can perform using CFEngine, explaining each one of them in detail. Through this chapter you will see many examples that you can (hopefully) use as-they-are for performing some real tasks, but you will also learn the underlying concepts that will be useful for adapting those examples, and for coming up with your own CFEngine policies.

Chapter 5 provides an overview of the CFEngine Design Center, a great resource containing ready-to-use components that allow you to perform a wide variety of tasks using CFEngine in an entirely data-driven manner, without having to be an expert in the CFEngine policy language.

Chapter 6 summarizes some generic tricks and patterns that you can use in CFEngine to achieve certain results. These are not specific recipes, but rather more generic techniques that you should learn to adapt and use in your own policies.

Finally, in Chapter 7 we will explore some topics that you may not need right away, but that will make your life easier in the future: maintaining separate CFEngine environments (for example, for development, testing, and production), testing mechanisms for CFEngine, using CFEngine with Vagrant, and managing services with CFEngine.

In Appendix A, contributed by Ted Zlatanov, you will find a detailed explanation of how to use Emacs to edit CFEngine policy files. Ted is the author and maintainer of cfengine-mode for Emacs.

In a similar vein, Appendix B, contributed by Neil H. Watson, describes how to use the Vim editor to manipulate CFEngine policy files. Neil is the author and maintainer of the vim_cf3 plugin for VIM.

Finally, in Appendix C, Aleksey Tsalolikhin gives us a summary of all the attributes that can be used with the CFEngine agent. Looking through this list provides a fantastic overview of the capabilities of CFEngine, and may even give you ideas for things to do on your systems.

As you read through the book, I encourage you to try out the examples. Preferably type them in yourself! I have learned from experience that typing the code (rather than downloading or copy/pasting it) helps tremendously to better understand a new language. It lets you develop a feeling for the code, it lets you make mistakes and figure out how to fix them, and it makes it easier to experiment and modify the examples. If you definitely don’t have the time or inclination to type them, you can download all the examples in this book from http://cf-learn.info/code.html, either individually or as a whole.

Online Resources

You can find the web page for this book at http://cf-learn.info/. In it you can find code samples, errata, a discussion forum, a CFEngine-related blog, and many other resources that you may find useful. I encourage you to visit, and of course to participate in the forum with suggestions, comments, or any other feedback.

If you are reading an electronic version of this book, you will find that most CFEngine keywords in the text, and some other concepts, are links that will take you to the corresponding part of the CFEngine Reference Manual.

You can find me on Twitter at http://twitter.com/zzamboni, and book-related tweets at http://twitter.com/LearningCF3.

You will find references to many other CFEngine-related resources in CFEngine Information Resources.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, CFEngine bundle and body names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

Tip

This icon signifies a tip, suggestion, or general note.

Caution

This icon indicates a warning or caution.

Using Code Examples

This book is here to help you get your job done. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Learning CFEngine 3 by Diego Zamboni (O’Reilly). Copyright 2012 Diego Zamboni, 9781449312206.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [e-mail].

Safari® Books Online

Note

Safari Books Online (www.safaribooksonline.com) is an on-demand digital library that delivers expert content in both book and video form from the world’s leading authors in technology and business.

Technology professionals, software developers, web designers, and business and creative professionals use Safari Books Online as their primary resource for research, problem solving, learning, and certification training.

Safari Books Online offers a range of product mixes and pricing programs for organizations, government agencies, and individuals. Subscribers have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technology, and dozens more. For more information about Safari Books Online, please visit us online.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

O’Reilly Media, Inc.

1005 Gravenstein Highway North

Sebastopol, CA 95472

800-998-9938 (in the United States or Canada)

707-829-0515 (international or local)

707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at:

http://bit.ly/learning_cfengine_3

You can also find many resources, including all the code samples, at the author’s web page for the book, which you can access at:

http://cf-learn.info/

To comment or ask technical questions about this book, visit the discussion forum at http://cf-learn.info/discussion.html, or send email to:

[e-mail]

For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia

Content Updates

September 2013 (third release)

• The book has been updated for CFEngine 3.5.2, the latest stable release as of this writing. This includes many new features, such as the new edit_template mechanism, improved services: promises, lots of new functions, and many others.

• The brand-new Chapter 5 provides an overview of the CFEngine Design Center. The Design Center had been mentioned in previous releases, but now is a mature and capable resource including powerful tools that allow you to use CFEngine in a completely new way, using entirely data-driven configuration. This chapter will get you started with the Design Center, and also provide pointers for you to explore on your own and even to contribute to it!

• Several new sections have been added, including Dynamic Loading and Execution, Using Vagrant with CFEngine, Service Management Using CFEngine, and several others.

• The brand-new Appendix C, contributed by Aleksey Tsalolikhin, contains a summary of all the attributes available for promises in agent bundles. Just browsing through this Appendix will give you an excellent overview of all the things you can do with CFEngine. In the electronic version of this book, each attribute links to its reference documentation in the CFEngine website, so you can look up the full details in seconds.

• Fixes to all known errata, typos, failures of clarity, and numerous other things.

November 2012 (second release)

• The book has been updated to reflect changes introduced up to CFEngine 3.3.9, including many features that were previously only available in CFEngine Enterprise.

• The installation instructions now reflect the availability of binary package repositories for many Linux distributions, as well as the “Free 25” Enterprise packages that allow you to try CFEngine Enterprise for free, for up to 25 machines.

• Vim fans rejoice! A new appendix, contributed by Neil H. Watson, provides details on how to use Vim to edit CFEngine policy files.

• This release fixes all known errata, fixing many typos, omissions, and other numerous miscellaneous things.

Acknowledgments

There were a lot of people who helped in the making of this book. I would like to thank my editor at O’Reilly Media, Andy Oram, who guided me and helped me through the process, and has continued to expertly do so for each update of the book. After working with him, I know why O’Reilly books are so good. Beyond simply providing editorial advice, he immersed himself in the topic, researched and learned it, asked me hard questions, and pointed me to interesting resources. His friendly but firm guidance has kept me going and made it possible for me finish this book.

I would like to thank my technical reviewers, Mark Burgess (all releases), Jesse Becker (first release), and Nick Anderson (third release), for their insightful and useful feedback. Their comments ranged from details about wording or the indentation of the examples, to high-level conceptual observations that made me rethink the focus of entire sections of the book. Their commentary made this book vastly better than it was before. Mark is also the original author of CFEngine, so without him and his work this book would not exist at all.

Halfway through writing this book (and partly as a result of it) I started a new job at CFEngine AS, the company behind CFEngine. I could not have found a better work environment, nor a more motivated and talented group of colleagues. They provided encouragement, feedback and useful discussions. In the first release of this book I was able to list the entire company in my acknowledgments. Since then the company has grown so much that this is no longer possible, but I continue to be indebted to each and every one of my colleagues for giving me an amazing work environment, for their support, encouragement and friendship, for teaching me so much about CFEngine, and for providing me with so many ideas, questions, and hours of interesting discussion.

CFEngine has an amazing and active user community, and working with such a community has always been a pleasure and an incredible learning experience. I would like to thank Aleksey Tsalolikhin (who kindly gave me permission to use his WordPress-installation policy in Chapter 4, and who contributed Appendix C), Ted Zlatanov (who maintains the excellent cfengine-mode for Emacs, and who contributed Appendix A), Neil Watson (who contributed Appendix B, and whose writing and posts have taught me so much about CFEngine), Nick Anderson^[1] (who has given me so much excellent feedback and encouragement, and tech-reviewed the third release of this book), Mike Svoboda (who continues to impress me both with his CFEngine expertise and his willingness to share complex real-world CFEngine policies for everyone to use), Jesse Becker (who started http://cfengineers.org/, and tech-reviewed the first release of this book), Loïc Pefferkorn (who tech-reviewed the third release of this book), Ben Bomgardner, Marco Marongiu, Seva Gluschenko, Nicolas Charles, Jonathan Clarke, Juliano Martinez, Bas Van Der Vlies, and many others too numerous to mention.

I would like to offer a special mention to the staff at O’Reilly Media, who made my life as an author much easier by always providing friendly and competent support and information. In particular I need to mention Sanders Kleinfeld, who expertly helped me understand and set up the syntax highlighting used in the electronic versions of this book (and which I think greatly enhances the readability of examples).

This book started life during November 2010 in the “Pragmatic Programmers Writing Month” or PragProWriMo. This is an event designed to mimic the well known “NaNoWriMo”, but for technical books. For one month, I committed to writing two pages every day, and from this effort the very first draft of this book was born. During this process I had the support and encouragement of a wonderful group of people, including Susannah Pfalzer, Michael Swaine, Travis Swicegood, Raymond Yee, and Bob Cochran. I also used http://750words.com/, a wonderful tool for writers created by Buster Benson and which helped me stay motivated throughout the month.

And of course, my life and work would not be the same without my family. My wife Susana has provided me with love, inspiration and encouragement, not to mention that, being also a sysadmin, she gave me some expert feedback on the book from the point of view of its target audience. And our two beautiful daughters Karina and Fabiola have, as always, been the joy of my life and a constant source of amazement and happiness. They all endured me spending many nights, weekends and off-hours working on “the book,” while keeping me sane with their love and support. Gracias mis bellas.

---

^[1] Both Ted and Nick are now my colleagues at CFEngine, which has given me the opportunity to work much more closely with them and to have even better access to their friendship and expertise.