Crafting the InfoSec Playbook

Preface

If you are reading this, chances are you are looking to enhance your threat detection capabilities and techniques, and up your game as an InfoSec, incident response, and network defender or manager. Threats both in scale, complexity, and profile have evolved dramatically over the last several years and continue to increase. Proper detection and response require a lot more effort and sophistication to remain effective. Building, maturing, and maintaining an effective incident response team is no easy task. We have talked with hundreds of security teams of all types and sizes who are waging the same war between the attackers and their organizations’ networks, users, and information. Few have done it well, but with a solid strategy, the right expertise, and the right infrastructure, you can compete with the bad guys.

Any good attacker will tell you—your expensive security monitoring and prevention tools are not enough to keep you secure. Successful computer security incident response teams (CSIRTs) realize that intrusions are inevitable, and the best plan is a combination of cultivated threat intelligence, vigilant monitoring for early detection, and rapid and thorough response. Having the right data available in the right tools doesn’t mean that the right people are looking at it and responding properly. Operational experience is invaluable and cannot be replaced by a magic black box or a single threat feed.

Our strategy focuses on collecting, organizing, mining, enhancing, and analyzing as many relevant data sources as possible in the hunt for intrusions and security breaches. We call our strategy, this bundle of detection and response methods, the playbook. We have developed a fundamental approach to building a successful incident response program that will detect the inevitable security incidents, minimize damage, yield enough information to share with the incident response community, and prevent successful attacks from recurring.

This book demonstrates how to boil down complex security monitoring, incident response, and threat analysis ideas into their most basic elements. Using a data-centric approach, we share how to create or refine your own unique incident detection strategy, how to keep your ideas and methods fresh, how to discover and develop your own threat intelligence, and how to compete against the malicious actors already attacking your network.

Should You Read This Book?

This book is for IT and information security (InfoSec) professionals, particularly incident or emergency response teams, InfoSec managers or directors, and IT architects, who want to either develop a nascent security monitoring and incident response program or evolve their existing program to a modern, more effective approach.

We wrote this book with InfoSec and incident response teams in mind, yet concepts such as log and data mining using a metadata-centric approach can certainly be applied to other fields as well, including system administration, threat research, and other data analytics. In the end, it is a strategy for organizing data, developing the right questions to ask, searching through the data, and then responding. Each chapter includes our observations and advice, based on real incidents and evidence, on how you can create a successful incident detection system.

This book will help you to answer common questions:

• How do I find bad actors on my network?

• How do I find persistent attackers?

• How can I deal with the pervasive malware threat?

• How do I detect system compromises?

• How do I find an owner or responsible parties for systems under my protection?

• How can I practically use and develop threat intelligence?

• How can I possibly manage all my log data from all my systems?

• How will I benefit from increased logging—and not drown in all the noise?

• How can I use metadata for detection?

Why We Wrote This Book

We wrote this book to help security professionals develop a unique and custom methodology, including broad data analysis and metadata extraction. Many of the basic concepts within incident response haven’t changed over the years. However, our do-it-yourself technology and data-centric approach is unique, and has evolved to compete with today’s extant threats. We’ve discovered and discuss the principal ideas that any team can automate high-fidelity security incident and breach detection with technology and preparation, as well as using basic information science to inform the human analyst for everything else. We stress the importance of investment in human intelligence and analytical skills. Effective and modern security monitoring requires metadata analysis, data organization, and information retrieval.

We’ve read plenty of InfoSec books. Generally, most have a few core ideas, and in some cases, some interesting and novel approaches. Yet many tend to fall into the same trap of spending page after page describing how to configure open source security software packages, or drone on about various configuration options replete with screenshots. Although inescapable, this book does describe some of the toolkits available for modern incident response, but the focus remains on strategy, technique, and informed decision making. We expect that readers already have some of their favorite tools deployed, and have some experience doing incident detection. We wrote this book to give those in the know, as well as those just getting started, practical advice and examples of not just how to install and configure tools, but how to strategically use them in real-world settings.

Cut to the Chase

Everyone wants to know how to find “bad stuff.” We’ve had many discussions with a diverse set of incident response teams around the world, and it’s clear there’s a need within the industry to formalize the methods to discover malicious hacking and policy violations in a structured and organized way.

In our day jobs for Cisco Systems’ world-class incident response team, we actively plan, deploy, and develop monitoring strategies and incident response techniques for many unique networks globally. We have formalized our approach and made it generic, yet applicable enough that we know we can teach other organizations how to best build their own playbooks while being specific enough to solve real-world problems. It’s also important to note that each organization may face different types of threats that may not be covered in our team’s specific playbook (the healthcare industry, for example, has substantially different concerns than we do in information technology). Therefore, it’s clear that a methodical and tested approach is what people need to hear.

It seems like there’s a product for every possible aspect of computer and network security. For years, security engineers have been promised and sold “silver bullet” security solutions that “correlate” all their events, and their security problems are “solved.” In our experience, these solutions often fall short of providing long-term value. We believe we have a solid approach that we will lay out in detail for any InfoSec professional.

How to Navigate This Book

Generically, the concepts cover the basic ideas of what a security incident response team should do, along with a well-reasoned approached for how they can do it best. The book attempts to cover these aspects in a strictly technology-agnostic manner so that regardless of any technology investments already in place, an organization can pick up the book and apply its principles to their own infrastructure.

If you are new to incident response and unfamiliar with how to build an effective monitoring program, begin with Chapter 1. If you’re missing out on some fundamental concepts (like understanding incident response and security monitoring), start with Chapter 2. If you are a salty InfoSec or incident response veteran like us, you can probably jump in at Chapter 4.

In any case, we’ve laid the book out as follows, for those nonlinear types:

• Chapter 1 introduces the incident response fundamentals: why it’s important to get back to basics, and how an understanding of the classic incident response model will inform your detection strategy.

• Chapters 2 and 3 help you to understand and answer the fundamental questions: What should you be protecting? What are the threats you face?

• Chapter 4 introduces the data-centric approach to security monitoring. It details how to work with the data you’re collecting and how to understand and use metadata.

• Chapter 5 discusses how to develop and structure incident detection logic into your own playbook.

• Chapter 6 takes the data-centric concept theory from Chapter 4 and turns it into operational practice. Additionally, it details how to effectively use your human resources, improve and demonstrate efficiency, and build the required systems to make your plan work.

• Chapter 7 details the different types of tools and technology available for security monitoring and incident detection, options for threat intelligence consumption and management, and how best to select and implement them.

• Chapters 8 and 9 detail techniques and strategies for developing detection logic using queries to put your data to work. These chapters build on the previous chapters and get into the core of developing your own threat detection. This chapter is all about plowing through the data to get the most out of your tools.

• Chapter 10 addresses the response phase of the incident response cycle, and what actions to take when events fire.

• Chapter 11 closes the book with a discussion of keeping your incident response plan and playbook relevant, and the challenges faced by next-generation network and host security.

Additional Resources

This book builds upon and draws inspiration from the previous literature discussing log management, InfoSec, incident response, and network security monitoring. Our recommended reading includes:

• Anton A. Chuvakin, Kevin J. Schmidt, and Christopher Phillips, Logging and Log Management (Waltham, MA: Syngress, 2013).

• Richard Bejtlich, Practice of Network Security Monitoring: Understanding Incident Detection and Response (San Francisco: No Starch Press, 2013).

• Chris Fry and Martin Nystrom, Security Monitoring: Proven Methods for Incident Detection on Enterprise Networks (Sebastopol, CA: O’Reilly, 2009).

• Richard Bejtlich, Extrusion Detection: Security Monitoring for Internal Intrusions (Boston, MA: Addison-Wesley Professional, 2005).

• Kenneth R. van Wyk and Richard Forno, Incident Response (Sebastopol, CA: O’Reilly, 2001).

• Karen Kent and Murugiah Souppaya, NIST 800-92: Guide to Computer Security Log Management.

• Chris Sanders, and Jason Smith, Applied Network Security Monitoring (Waltham, MA: Syngress, 2013).

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

Note

This icon signifies a tip, suggestion, or general note.

Warning

This icon indicates a warning or caution.

Safari^® Books Online

Safari Books Online is an on-demand digital library that delivers expert content in both book and video form from the world’s leading authors in technology and business.

Technology professionals, software developers, web designers, and business and creative professionals use Safari Books Online as their primary resource for research, problem solving, learning, and certification training.

Safari Books Online offers a range of plans and pricing for enterprise, government, education, and individuals.

Members have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like Maker Media, O’Reilly Media, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technology, and hundreds more. For more information about Safari Books Online, please visit us online.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

• O’Reilly Media, Inc.
• 1005 Gravenstein Highway North
• Sebastopol, CA 95472
• 800-998-9938 (in the United States or Canada)
• 707-829-0515 (international or local)
• 707-829-0104 (fax)

To comment or ask technical questions about this book, send email to: [e-mail].

For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments

It took a lot of effort from a lot of talented people to pull off our playbook at Cisco. The entire CSIRT works in some way or another with the playbook or its supporting systems, although we’d specifically like to extend thanks to the following people from CSIRT present and past for their explicit support and collaboration: Gavin Reid, Michael Scheck, Chris Fry, Martin Nystrom, Lawrence Dsouza, Dustin Schieber, Ray Espinoza, James Sheppard, Joseph McCauley, David Schwartzburg, Imran Islam, Tammy Nguyen, Jayson Mondala, Darryl Delacruz, Marianela Morales, Juan Gabriel Arce, Julian Umana, Ashwin Patil, Archana Mendon, and Chad Ruhle.

The incredibly talented technical reviewers and editors took our incomplete thoughts, ill-defined phrases, and occasionally our taboo and embarrassing grammar failures and turned them around on us. Their insight and help really took this book to the next level. A hearty thanks to everyone: Devin Hilldale, Sonya Badigian, Chris Fry, Scott McIntyre, Matt Carothers, Seth Hanford, and Robert Sheehy.

A very special thank you to the folks at O’Reilly for agreeing to let us write a book for them, and helping us turn our thoughts and writings into a cohesive and comprehensive package. Many thanks to Mike Loukides, Amy Jollymore, Katie Schooling, Rebecca Demarest, Jasmine Kwityn, Kristen Brown, and Dan Fauxsmith.