Web Security Testing Cookbook

Preface

Web applications are everywhere and in every industry. From retail to banking to human resources to gambling, everything is on the Web. Everything from trivial personal blogs to mission-critical financial applications is built on some kind of web application now. If we are going to successfully move applications to the Web and build new ones on the Web, we must be able to test those applications effectively. Gone are the days when functional testing was sufficient, however. Today, web applications face an omnipresent and ever-growing security threat from hackers, insiders, criminals, and others.

This book is about how we test web applications, especially with an eye toward security. We are developers, testers, architects, quality managers, and consultants who need to test web software. Regardless of what quality or development methodology we follow, the addition of security to our test agenda requires a new way of approaching testing. We also need specialized tools that facilitate security testing. Throughout the recipes in this book, we’ll be leveraging the homogenous nature of web applications. Wherever we can we will take advantage of things that we know are uniformly true, or frequently true, about web applications. This commonality makes the recipes in this book versatile and likely to work for you. Moreover, it means that you will develop versatile testing tools that are likely capable of testing more than just one application.

Who This Book Is For

This book is targeted at mainstream developers and testers, not security specialists. Anyone involved in the development of web applications should find something of value in this book. Developers who are responsible for writing unit tests for their components will appreciate the way that these tools can be precisely focused on a single page, feature, or form. QA engineers who must test whole web applications will be especially interested in the automation and development of test cases that can easily become parts of regression suites. The recipes in this book predominantly leverage free tools, making them easy to adopt without submitting a purchase requisition or investing a significant amount of money along with your effort.

The tools we have selected for this book and the tasks we have selected as our recipes are platform agnostic. This means two very important things: they will run on your desktop computer no matter what that computer runs (Windows, MacOS, Linux, etc.), and they will also work with your web application no matter what technology your application is built with. They apply equally well to ASP, PHP, CGI, Java, and any other web technology. In some cases, we will call out tasks that are specific to an environment, but generally that is a bonus, not the focus of a recipe. Thus, the audience for this book can be any developer or tester on any web platform. You do not need special tools (except the free ones we discuss in this book) or special circumstances to take advantage of these techniques.

Leveraging Free Tools

There are many free testing tools that can be used to help a developer or a tester test the fundamental functions of their application for security. Not only are these tools free, but they tend to be highly customizable and very flexible. In security, perhaps more than in any other specialized discipline within QA, the best tools tend to be free. Even in the network security field, where commercial tools now are mature and powerful, it was a long time before commercial tools competed with readily available, free tools. Even now, no network assessor does his job strictly with commercial tools. The free ones still serve niche roles really well.

In so many cases, however, free tools lack documentation. That’s one of the gaps that this book fills: showing you how to make good use of tools that you might have heard of that don’t have good documentation on the how and why of using them. We think mainstream developers and testers are missing out on the promise of free and readily available tools because they do not know how to use them.

Another barrier to effectively testing web applications with free tools is a general lack of knowledge around how the tools can be put together to perform good security tests. It’s one thing to know that TamperData lets you bypass client-side checks. It’s another thing to develop a good cross-site scripting test using TamperData. We want to get you beyond making good web application tests and into making good security test cases and getting reliable results from those tests.

Finally, since many development and QA organizations do not have large tool and training budgets, the emphasis on free tools means that you can try these recipes out without having to get a demo license for an expensive tool.

About the Cover

The bird on the cover is a nutcracker (Nucifraga columbiana) and it makes an excellent mascot for the process of security testing web applications. Nutcrackers try to pry open unripe pine cones to extract the seeds. Their beaks are designed to go into those small nooks and crannies to get the food out. As security testers we are trying to use specialized tools to pry open applications and get at private data, privileged functions, and undesired behavior inside. One of the roles of this book is to give you lots of specialized tools to use, and another is to hint at the nooks and crannies where the bugs are hidden.

The nutcracker is also remarkable in its ability to remember and return to all the different places that it has hidden food. It stores the seeds it has gathered in hundreds or thousands of caches, and then it comes back and gets them throughout the winter. Our testing activities parallel the nutcracker again because we build up batteries of regression tests that record the places we historically have found vulnerabilities in our application. Ideally, using the tools and techniques in this book, we’ll be revisiting problems that we found before and making sure those problems are gone and stay gone.

For more information on Nucifraga columbiana, see The Birds of North America Online from Cornell University at http://bna.birds.cornell.edu/bna/. For more information on web application security testing, read on.

Organization

The book divides material into three sections. The first section covers setting up tools and some of the basics concepts we’ll use to develop tests. The second section focuses on various ways to bypass client-side input validation for various purposes (SQL injection, cross-site scripting, manipulating hidden form fields, etc.). The final section focuses on the session, finding session identifiers, analyzing how predictable they are, and manipulating them with tools.

Each recipe will follow a common format, stating the problem to be solved, the tools and techniques required, test procedure, and examples. Recipes will share a common overall goal of fitting into a testing role. That is, you will be interested in the recipe because it makes it easier to test some security aspect of your web application.

The book is organized overall from basic tasks to more complex tasks, and each major section begins with relatively simple tasks and gradually builds to more complex tasks. The first recipes are simply eye-opening exercises that show what happens behind the scenes in web applications. The final recipes put many building blocks together into complex tasks that can form the basis of major web application security tests.

Section One: Basics

We begin by getting your test environment set up. This section familiarizes you with the foundations you will use throughout the book. The first thing you need to learn is how to get tools set up, installed, and operational. Then you need to understand the common features of web applications that we will be using to make our tests as broadly applicable as possible.

Chapter 1, Introduction, gives you our vision for software security testing and how it applies to web applications. There’s a little terminology and some important testing concepts that we will refer to throughout the book.

Chapter 2, Installing Some Free Tools, includes a whole toolbox of different, free tools you can download and install. Each includes some basic instructions on where to find it, install it, and get it running. We will use these tools later in the recipes for actually conducting security tests.

Chapter 3, Basic Observation, teaches you the basics of observing your web application and getting behind the façade to test the functionality of the system. You will need these basic skills in order to do the more advanced recipes later in the book.

Chapter 4, Web-Oriented Data Encoding, shows a variety of data encodings. You need to know how to encode and decode data in the various ways that web applications use it. In addition to encoding and decoding, you need to be able to eyeball encoded data and have some idea how it has been encoded. You’ll need to decode, manipulate, and reencode to conduct some of our tests.

Section Two: Testing Techniques

The middle section of the cookbook gives you some fundamental testing techniques. We show you both manual- and bulk-scanning techniques. The chapters cover both general tools as well as specific tools to do a variety of different jobs that you’ll combine into more complex tests.

Chapter 5, Tampering with Input, discusses the most important basic technique: malicious input. How do you get it into your application? How can you look at what’s happening in the browser and what it’s sending to the web application?

Chapter 6, Automated Bulk Scanning, introduces several bulk-scanning techniques and tools. We show you how to spider your application to find input points and pages, as well as ways to conduct batch tests on some specialized applications.

Chapter 7, Automating Specific Tasks with cURL, shows you a great tool for building automated tests: cURL. We introduce a few obvious ways to submit batches of tests, gradually progress to harder tasks such as retaining state when you log in and manipulating cookies, and ultimately build up to a complex task: logging in on eBay.

Chapter 8, Automating with LibWWWPerl, is focused on Perl and its LibWWWPerl (LWP) library. It’s not a book on how to program Perl. It’s a set of specific techniques that you can use with Perl and the LWP library to do interesting security tests, including uploading viruses to your application, trying out ridiculously long filenames, and parsing the responses from your application. It culminates in a script that can edit a Wikipedia web page.

Section Three: Advanced Techniques

The advanced techniques in the final chapters build on the recipes earlier in the book. We combine them in ways that accomplish more tests or perhaps address security tests that were not demonstrated in earlier recipes.

Chapter 9, Seeking Design Flaws, discusses the unintentional interactions in your web application and how you can reveal them with good security tests. The recipes in this chapter focus on ways we can enable tests with our testing programs we’d never be able to do otherwise. This includes predictable identifiers, weak randomness, and repeatable transactions.

Chapter 10, Attacking AJAX, shows you a lot of the top web attacks and how you can execute them in a systematic, test-focused way using the techniques we’ve taught earlier. Injecting Server-Side Includes (SSI), abusing LDAP, and SQL injection are a few of the attacks discussed in Chapter 10.

Chapter 11, Manipulating Sessions, looks at AJAX, a technology that predominates so-called Web 2.0 applications. We show you how to get behind the scenes on AJAX and test it both manually and automatically. We intercept client-side requests to test server-side logic and vice versa, testing the client-side code by manipulating the server’s responses.

Chapter 12, Multifaceted Tests, focuses on sessions, session management, and how your security tests can attack it. It gives you several recipes that show you how to find, analyze, and ultimately test the strength of session management.

Conventions Used in This Book

When we refer to Unix-style scripts or commands, we use both typography and common Unix documentation conventions to give you additional information in the text. When we refer to Windows-oriented scripts or commands, we use typography and documentation conventions that should be familiar to Windows users.

Typographic Conventions

Plain text

Indicates menu titles, menu options, menu buttons, and keyboard accelerators (such as Alt and Ctrl).

Italic

Indicates new or technical terms, system calls, URLs, hostnames, email addresses.

Constant width

Indicates commands, options, switches, variables, attributes, keys, functions, types, objects, HTML tags, macros, the contents of files, or the output from commands, filenames, file extensions, pathnames, and directories.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values.

Note

This icon signifies a tip, suggestion, or general note.

Warning

This icon indicates a warning or caution.

There are times when it is very important to pay attention to the typography because it distinguishes between two similar, but different concepts. For example, we often use URLs in our solutions. Most of the time the URL is fictitious or is the official example URL for the Internet: http://www.example.com/. Notice the difference between the constant width typeface of that URL and the typeface of http://ha.ckers.org/xss.html, a website that has many cross-site scripting examples. The former is not a URL you should actually visit. (There’s nothing there anyways). That latter is a useful resource and is intended to be a reference for you.

Conventions in Examples

You will see two different prompts in the examples we give for running commands. We follow the time-honored Unix convention of using % to represent a non-root shell (e.g., one running as your normal userid) and # to represent a root-equivalent shell. Commands that appear after a % prompt can (and probably should) be run by an unprivileged user. Commands that appear after a # prompt must be run with root privileges. Example 1, shows four different commands that illustrate this point.

Example 1. Several commands with different prompts

% ls -lo /var/log
% sudo ifconfig lo0 127.0.0.2 netmask 255.255.255.255
# shutdown -r now
C:\> ipconfig /renew /all

The ls command runs as a normal user. The ifconfig command runs as root, but only because a normal user uses sudo to elevate his privileges momentarily. The last command shows the # prompt, assuming that you have already become root somehow before executing the shutdown command.

Within Windows we assume you can launch a CMD.EXE command prompt as necessary and run commands. The ipconfig command in Example 1 shows what a typical Windows command looks like in our examples.

Using Code Examples

This book is here to help you get your job done. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Web Security Testing Cookbook by Paco Hope and Ben Walther. Copyright 2009 Brian Hope and Ben Walther, 978-0-596-51483-9.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [e-mail].

Safari® Books Online

Note

When you see a Safari® Online icon on the cover of your favorite technology book, that means the book is available online through the O’Reilly Network Safari Bookshelf.

Safari offers a solution that’s better than e-books. It’s a virtual library that lets you easily search thousands of top tech books, cut and paste code samples, download chapters, and find quick answers when you need the most accurate, current information. Try it for free at http://safari.oreilly.com.

Comments and Questions

Please address comments and questions concerning this book to the publisher:

O’Reilly Media, Inc.

1005 Gravenstein Highway North

Sebastopol, CA 95472

800-998-9938 (in the United States or Canada)

707-829-0515 (international or local)

707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at:

http://www.oreilly.com/catalog/9780596514839

To comment or ask technical questions about this book, send email to:

[e-mail]

For more information about our books, conferences, Resource Centers, and the O’Reilly Network, see our website at:

http://www.oreilly.com

Acknowledgments

Many people helped make this book possible, some of them in big ways and others in critical, yet nearly invisible ways. We’d like to acknowledge them here.

Paco Hope

No man is an island, least of all me. This book could not come to be without the help and inspiration from a lot of people. First and foremost I thank my wife, Rebecca, who administered everything that doesn’t run Mac OS (like children, houses, and pets). She is the master of handling bad input, unexpected output, and buffer overflows.

I thank both my colleagues and customers at Cigital, Inc. for introducing me to risk-based approaches to software security, quality, and testing. Many Cigitalites have had a lasting impact on my approach to software security and testing. Here are a few in reverse alphabetical order (because John always ends up last): John Steven, Amit Sethi, Penny Parkinson, Jeff Payne, Scott Matsumoto, Gary McGraw, and Will Kruse. Thanks to Alison Wade and the great folks at Software Quality Engineering (SQE) for the opportunity to speak at their software quality events and meet amazing professionals who are dedicated to their craft. A quick thank you to Bruce Potter who helped me get started writing; he rocks.

Ben Walther

Paco Hope had the vision, the gumption, the contacts, and was the driving force behind this book. The chapters that don’t read like a textbook? Those are his. Thanks, Paco, for the carrots and sticks, writing, and technical advice.

My colleagues at Cigital, thank you for your guidance, teaching, and good humor—particularly about all those office pranks.

Lastly, anyone reading this has my admiration. Continual learning is one of the highest ideals in my life—that you’d take your time to expand your knowledge speaks very highly of your professional and personal principles. I welcome conversation and comment on anything in this book (particularly if you can show me a thing or two)—email me at [e-mail]. Or, leave a comment on my blog at http://blog.benwalther.net.

Our Reviewers

We appreciate all the feedback we received from our technical reviewers. They definitely kept us on our toes and made this book better by lending their expert advice and opinions. Thanks to Mike Andrews, Jeremy Epstein, Matt Fisher, and Karen N. Johnson.

O’Reilly

Finally, we thank the staff at O’Reilly, especially Mike Loukides, Adam Witwer, Keith Fahlgren, and the hoards of talented individuals who helped make this book a reality. Without Adam’s DocBook wizardry and Keith’s Subversion heroics, this book would have been a tattered bunch of ones and zeros.