AWS System Administration

Preface

System administration is a complicated topic that requires practitioners to be familiar with an ever-expanding range of applications and services. In some ways, Amazon Web Services (AWS) is just another tool to add to your toolkit, yet it can also be considered a discipline in and of itself. Successfully building and deploying infrastructure on AWS involves a thorough understanding of the underlying operating system concerns, software architecture, and delivery practices, as well as the myriad components that make up Amazon Web Services.

Mike runs a DevOps consultancy, helping startups and small businesses reap the benefits of tools and processes that were previously available only to organizations with large teams of system administrators. Many of these businesses do not have a dedicated system administrator, and the development team is responsible for deploying and maintaining the architecture.

In working with these clients, Mike noticed patterns in how people were working with AWS. Those who came from a pure development background, without any sysadmin experience, would often build an infrastructure that left out many of the things sysadmins would take for granted, such as monitoring and logging. The lack of monitoring and logging would then make it difficult to track down issues, leading to more downtime than necessary.

At the other end of the spectrum were clients with a lot of sysadmin experience, but less or no development experience. This group was more likely to treat AWS as nothing more than a virtual machine hosting provider, simply using EC2 to run a fleet of static instances without taking advantage of any high-availability features such as Auto Scaling Groups and Elastic Load Balancing. This is akin to buying a Ferrari and then using it only to buy groceries once per week: fun, but not very cost-effective.

Using AWS requires a fundamentally different mindset than when deploying groups of static servers. You do not simply set up some servers and then periodically perform maintenance. Instead, you use the full AWS toolset (automatic instance replacement, scaling up and down in response to demand, etc.) to build a system. In this sense, it is more like programming than traditional system administration.

Federico’s work as the lead Product Manager for Ubuntu Server at Canonical placed him on the front lines from the vendor perspective. Looking at how users and public cloud vendors were integrating Ubuntu into their infrastructure informed the team’s decisions as to what user problems to solve first, and led to the creation of Canonical’s Certified Public Cloud program, which solves integration problems for public cloud vendors and directly manages the relationship between Ubuntu and the Amazon AWS technical team.

The Ubuntu Server team’s cloud-first focus led to the creation of technologies like Cloud-init, and produced the early, smooth integration of Ubuntu with public cloud that resulted in its popularity there today. Federico’s aim has been to complement Mike’s knowledge as a power user of public cloud with his behind-the-scenes insight into where things can go wrong for users.

The aim of this book is to help you reach a balance between development and operational focus, and help you make the right choice for your application’s specific hosting requirements. If you are a developer, this book will give you enough system administration knowledge to ensure that you are using AWS effectively, and help you build a robust and resilient application infrastructure. For system administrators, it will show you how you can keep your favorite tools and processes while working with AWS, and hopefully save you from reinventing some wheels along the way.

AWS is a collection of cloud computing services that can be combined to build scalable and reliable applications and services. It comprises a number of components, each with their own names and configuration options, which are offered under the AWS umbrella. Some of these—such as EC2 and S3—are extremely popular and well-known. Others, such as Kinesis and CloudFormation, are less well-known. Because covering each of these services in detail would result in a multivolume tome of formidable size, this book focuses on the more commonly used services and provides jumping-off points for learning about the others.

If you are familiar with AWS, feel free to hop between chapters to find the information that is most interesting or relevant to your current project. Beginners to AWS should work through the book sequentially, as each chapter builds on information presented in the previous chapters.

Chapter 1 helps you get set up with the tools you will need to interact with AWS and build the example infrastructure.

Chapter 2 introduces what is perhaps the most well-known of all AWS services, EC2. This chapter also introduces our favorite AWS service, CloudFormation.

In Chapter 3, we look at the most important security features provided by AWS, and how they combine with service functionality to enable secure operation.

Chapter 4 introduces configuration management tools, a common requirement when automating a cloud infrastructure. Using these tools, Chapters 5 and 6 demonstrate the process of deploying an example application to AWS.

Chapter 7 looks at some of the methods of deploying application and infrastructure updates to your environment. Chapter 8 builds on this and discusses the creation of reusable components to save time.

Log management, a more traditional sysadmin task that has some interesting implications in the cloud, is the topic of Chapter 9.

Chapter 10 covers another traditional sysadmin task: DNS with Amazon’s Route 53 service.

Monitoring with Amazon’s CloudWatch service and other monitoring tools is discussed in Chapter 11.

Finally, Chapter 12 looks at some of the ways of backing up your data both in and outside the Amazon cloud.

Audience

This book is written for system administrators and developers. We assume you are comfortable with the basic tools used to administer the operating system and common services such as DNS. If you plan to use Puppet or Chef for automation, you need to learn basic information about those tools elsewhere. You should have a working knowledge of Git or another source code management system. We do not expect you to have prior knowledge of AWS or other virtualization and cloud products.

Where an operating system choice was required, we tested our examples on Ubuntu 16.04 “Xenial” and Ubuntu 14.04 “Trusty.” For users on Amazon AWS today, Ubuntu is the most popular option, so we used it as both client environment and for our EC2 instances; examples will nonetheless run on most Linux distributions with minimal or no change. Whenever we used an RPM-based distribution, we checked our work against Amazon Linux, which remains the second most popular choice of distribution as we write.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, data types, and environment variables.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

Note

This icon signifies a general note.

Tip

This icon signifies a tip or suggestion.

Caution

This icon indicates a warning or caution.

Using Code Examples

This book is here to help you get your job done. Major examples can be downloaded from our GitHub repository. Many other small examples are scattered through the book; we have not bothered to include them in the repository because they are fairly easy to type in.

In general, you may use the code in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “AWS System Administration by Mike Ryan and Federico Lucifredi (O’Reilly). Copyright 2018 by Mike Ryan and Federico Lucifredi 978-1-449-34257-9.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [e-mail].

O’Reilly Safari

Note

Safari (formerly Safari Books Online) is a membership-based training and reference platform for enterprise, government, educators, and individuals.

Members have access to thousands of books, training videos, Learning Paths, interactive tutorials, and curated playlists from over 250 publishers, including O’Reilly Media, Harvard Business Review, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, and Course Technology, among others.

For more information, please visit http://oreilly.com/safari.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

• O’Reilly Media, Inc.
• 1005 Gravenstein Highway North
• Sebastopol, CA 95472
• 800-998-9938 (in the United States or Canada)
• 707-829-0515 (international or local)
• 707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://bit.ly/aws-system-administration.

To comment or ask technical questions about this book, send email to [e-mail].

For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments

This book would not exist were it not for the many amazing people who helped us along the way. Mike would like to thank his family for their love and support, which allowed him to become the geek he is today. He would also like to thank Cynthia Stolk for supplying endless cups of coffee to fuel this book, and Rachel Kersten and Rebecca Lay for making sure it actually got finished. Federico thanks his wife Irena for being his loving inspiration and Mark Shuttleworth for convincing him “this cloud thing” was really the top priority in a sea of top priorities.

Thanks are due to friends and colleagues who gave feedback and suggestions: Bartek Swedrowski, Dave Letorey, Guyon Morée, Jurg van Vliet, Keith Perhac, Peter van Kampen, Rick van Hattem, Ross Gynn, Sofie Pelmelay, and Thierry Schellenbach. Martin Guenthner, Jess Males, Ilan Rabinovitch, Douglas Knight, Kapil Thangavelu, Ben Howard, Peter Bowen, Kyle Bader, Stephen Walli, Olivier Engler, and William Ricker patiently reviewed many drafts, told us what the book lacked, and kept us honest. Either directly or indirectly, you helped shape this book.

Finally, we would like to thank the excellent team at O’Reilly for making this happen. Particular thanks are due to our editor, Andy Oram, for persevering on the long road. Nikki McDonald’s help was instrumental to getting this first edition past the finish line. And, of course, this book would be empty if not for the amazing team behind Amazon Web Services.