Agile Application Security

Chapter 4. Working with Your Existing Agile Life Cycle

So you want to start building more secure software, but your security and compliance checklists require you to have design review gates and penetration testing, and you can’t work out where these fit in an Agile life cycle.

Traditional Application Security Models

In a traditional application security model, the security touchpoints during software development are mostly gates that the product must stop at and pass. Some security work is done in parallel with development, and the gates are a chance to verify that the security work and development work haven’t diverged. The following are common security gates:

Design or requirements review

The security team looks at the requirements list or early designs and adds any security requirements based on threat models and attack trees.

Architecture review

The security team reviews the proposed architecture, such as infrastructure or information flows, and proposes a set of security controls in order to minimize risk.

Code review

The security team reviews sensitive areas of code and confirms that the security requirements are met, and that the code matches the architecture.

Security testing

The security team or an external team checks the test version of the product against a set of security requirements to see if the system is safe and secure.

The idea behind these gates is that work is delivered in large batches. It is predicated on the old rule that the earlier a defect is caught, the cheaper it is to fix; therefore we need to do a security review as early as possible to catch security defects before they get too far.

Agile practitioners argue that while this rule is broadly speaking true—catching a defect later is more expensive than catching one earlier—the solution is not to attempt the impossible task of catching all defects earlier, but instead to focus on reducing the cost of fixing defects by making change safer and easier.

The same is true of security features and controls. We want to reach a fine balance between finding and fixing (or better, preventing) security problems up front where it makes sense to do so, and making sure that we can fix them quickly and cheaply later if something gets by.

Some security defects fall into a special category of defects: design flaws that are critical showstoppers or fundamental issues with the way a system works. While many bugs are relatively easy to fix at a later date and may not drastically increase the risk to the system, correcting a fundamental security design flaw may require you to start again from scratch, or force people to fix an endless stream of security bugs one-by-one.

For example, choosing the wrong language or framework, or relying too much on features of a PaaS platform or on an infrastructure black box to take care of problems for you, can lead to serious security risks, as well as fundamental run time reliability and scalability problems.

Therefore, even in an Agile environment, security needs to be involved at the early product direction stages and in architecture discussions, not just in later parts of the development life cycle.

What’s the Difference Between a Bug and a Flaw?

While in common speech, the terms “bug” and “flaw” can be used interchangeably, in security, each term is used to describe very different types of security issues.

A bug is a low-level implementation failure that results in a system operating in an unintended manner that gives rise to a security issue. A generic example of a bug would be failing to sanitize user-supplied input that gets used in a database call, leading to SQL injection.

A famous example of a security bug would be the Heartbleed vulnerability in OpenSSL (CVE-2014-0160), where the server failed to check if the stated length of data sent in a heartbeat message was actually the length of the data sent. This mistake allowed an attacker to cause a vulnerable server to send him excess data, which leaked secrets.

A flaw is a design failure where a system operates exactly as intended but inherently causes a security issue to arise. Flaws often come from systems that have been designed without security needs in mind. A generic example of a flaw would be a client-server system that is designed to perform the authentication of a user on the client side and to send the server the result of the authentication check, making the system vulnerable to man-in-the-middle attacks.

A famous example of a security flaw would Dirty Cow (CVE-2016-5195), an issue that affected the GNU/Linux kernel. The design of the copy-on-write (COW) and memory management systems had a race condition which when triggered would allow an unprivileged user to write to a file owned by the root user, allowing that user to escalate her privileges to those of root.

Why are these distinctions important? In general, fixing a bug is cheaper and easier than fixing a flaw, as a bug usually only requires correcting a developer’s specific mistake, whereas a flaw can require a significant restructuring of both the problematic code as well as code that is reliant on that code. Restructuring code within an already implemented system can be highly complex and runs the risk of introducing new bugs and flaws.

In certain situations flaws are not fixable at all, and the system will remain vulnerable to security issues for the rest of its life. The best way to prevent security flaws from being introduced into an application is to ensure security is considered during its design. The earlier security architecture is included in the development life cycle, the fewer security flaws are likely to be designed in.

Let’s work backward from coding and testing (which are done simultaneously in Agile) to planning and design, and discuss what security activities should happen at each stage.

Per-Iteration Rituals

During a product development iteration, there are a number of rituals where security should be involved.

Who Is Security?

In an organization embracing Agile development, who exactly do we mean when we say “security”?

The answer is going to depend on the size of your company and the focus of your team. If you are a startup or small company, you may not have a dedicated security specialist available. Instead, someone on the team may have to own that role, with occasional guidance and checkups from an outside expert (however, see Chapter 14 for issues and risks with this model).

In a larger organization, you may have a dedicated specialist or team that owns security for the company. However, most security teams cover physical security, network security, and compliance and audit responsibilities, not necessarily application security advice and support.

Some organizations that take security and compliance seriously have a person on each Agile team who is dedicated to the security role for that team. This might be a job share, 20% of the time for example; or it might be a dedicated security person shared across multiple teams; or in some cases teams may have their own full-time security specialist.

The key thing is that somebody on the team needs to be able to take on the role of “security” during each iteration and ensure that security concepts and risks, and the perspectives of an attacker, are included in requirements, design, coding, testing, and implementation. Whoever takes on this responsibility, as well as the other members of the team, have to approach security as an enabler, and understand and agree that considering security early on and throughout the process contributes to a more successful end product.

At the daily stand-up meeting, where the state of stories is reviewed, the team should be listening for any issues raised that may affect security and privacy. If there are stories of security importance, then progress on those stories needs to be monitored.

During development, having someone with a strong security background available to pair on security-sensitive code can be worthwhile, especially on teams that follow pair programming as a general practice.

If your team does team code reviews (and it should) through pull requests or collaborative review platforms, then having a security person reviewing code changes can help identify areas of the code base that need careful attention and additional testing.

At the beginning of each iteration, at the kick-off or planning meeting, most teams walk through all the potential stories for the iteration together. Somebody representing security should be present to ensure that security requirements are understood and applicable to each story. This helps ensure that the team owns and understands the security implications of each story.

At the end of each iteration, there are times when security should also be involved in the reviews and retrospective meetings to help understand what the team has done and any challenges that it faced.

All of these points provide opportunities for security to engage with the development team, to help each other and learn from each other, and to build valuable personal connections.

Making the barrier to interacting with the security team as low as possible is key to ensuring that security does not get in the way of delivery. Security needs to provide quick and informal guidance, and answers to questions through instant messaging or chat platforms, email, and wherever possible in person so that security is not seen as a blocker. If those responsible for security are not reachable in an easy and timely manner, then it is almost certain that security considerations will be relegated to the sidelines as development charges ahead.

Tools Embedded in the Life Cycle

The best way to assure yourself of the security of the system you are building is to perform an exhaustive set of checks before it is allowed out.

Advances in security technology allow us to use tools such as the following:

• Gauntlt

• BDD-Security

• Snyk

• InSpec

• Brakeman

• ZAP

• OSQuery

• TruffleHog

• Dependency-Check

• Error-Prone

These tools and others that we will look at in this book automate many of the assurance processes that manual testers are traditionally responsible for. They don’t obviate the need for any manual testing, but they can help to prioritize time and effort in testing by removing the need to do detailed, routine, and time-consuming work, and ensuring that testing is more repeatable and reliable.

The security team needs to own these tools, while the development team owns the implementation of the tools in its pipeline.

This means that the development team cares about ensuring that the tool is in its pipeline, that it is correctly configured for the project, and that the team can act on the results.

The security team is responsible for deciding what features the tool should have, for making it easy to embed in the pipeline, and for ensuring that the tool or tools cover the areas that the team is most concerned about.

Pre-Iteration Involvement

Most Agile teams not only have a product development team working in iterations, but a product designer or design team working in advance of the development team, working on design problems, prototypes, and architectural discussions. The output of this team feeds directly into the product backlog, ensuring that the development team is primed with stories ready for the forthcoming iteration.

We’ve seen several ways this can work, from a separate design team working just one iteration ahead, to monthly product design meetings that produce several sprints’ worth of backlog in batches.

Security is critical in the design and architecture phase. It is at this point that instead of worrying about software library patch levels or secure coding guidelines, you need to be thinking about secure service design, trust modeling, and secure architecture patterns.

It’s important to note that design in this case is not about the look of the system—we aren’t talking about Photoshop jockeys here. Design is about how the system works, the principal user interactions, APIs, and the flow of data through the system.

The design team should have access to security training or security expertise to ensure that the service the team is designing enables security through the user experience. Examples of this work may include understanding how or whether to obfuscate user details when displayed, how changes to information are gathered, and what identification requirements are needed for specific actions.

The architecture team may also need access to a security architect for any complex architecture. Building an architecture to be secure by design is very different than writing code that is secure or ensuring that there are no defects in the product.

Architects need to think carefully about threat models (or threat reckons for teams that don’t do formal modeling) and about trust boundaries in their systems (something that we’ll explain later in Chapter 8, Threat Assessments and Understanding Attacks).

Tooling for Planning and Discovery

Security teams should be providing tooling, processes, and guidance that help product managers, architects, and developers follow good security practice while designing a new system.

This might be as simple as a wiki with common security patterns already in use in the organization, or it might be threat modeling tools, or technical risk assessment checklists or questionnaires that make it easy for architects to understand security problems and how to deal with them up front.

Post-Iteration Involvement

Agile teams that are adopting a DevOps culture need automated fast systems for building and deploying their work into production reliably and repeatably.

Security matters during the build and deployment process for several different reasons:

1. Providing assurance that the correct thing was built and deployed

2. Assuring that the thing that was built and deployed is secure

3. Ensuring that the thing will be built and deployed in a secure way every time

Security checks that happen at this stage need to be automatable, reliable, repeatable, and understandable for a team to adopt them.

Manual processes are the opposite of this: most are not reliable (in terms of consistently catching the same errors), repeatable (in terms of repeating a finding), or understandable to the team.

Ideally the security team is already heavily involved in operations: it has to help define business continuity plans, incident response plans, as well as monitor and audit suspicious activity on the systems.

But is that an effective use of the team’s time? The security team should know what features have been released in the last iteration and ensure that those new features are added to the logging, fraud detection, analysis, and other security systems.

It should be clear that high-risk features need to be monitored more closely. One possible action during iteration is to accept the risk temporarily, essentially assuming that the likelihood of the risk happening before a control can be put in place in a few iterations time is very low.

But the security team needs to be aware of these risks and monitor them until they are mitigated. This means that early on in system development, effective logging and auditing controls need to be put in place to ensure that we can see this kind of thing.

We’ll cover a lot more on this in Chapter 12 under security in operations.

Tools to Enable the Team

As well as automated security testing tools that can be easily plugged into developer workflows, the security team should look for ways to make the development team’s job easier, that help the team develop and deliver software faster—and at the same time, more securely.

For example, the security team can help development create effective build and deployment pipelines, and come up with a simple process and tools to compile, build, test, and automatically deploy the system in ways that also include security checks all along the path.

The security team may also want to provide tools for internal training, such as OWASP’s WebGoat Project, the Damn Vulnerable Web Services project, or other intentionally vulnerable applications that developers can explore and test, so that they can learn about how to find and remediate security issues safely.

The security team should do everything that it can to ensure that the easiest way to build something inside the organization is the safe and secure way, by providing teams with secure headers, hardened runtime configuration recipes and playbooks, and vetted third-party libraries and images that are free from vulnerabilities, which teams can grab and use right away. We’ll look at how to do this in later chapters of this book.

When security stops being the team that says no, and becomes the team that enables reliable code to ship, then that’s true Agile security.

Compliance and Audit Tools

How about once the system is in production? As well as tooling that does simple vulnerability testing, good security teams know that they need to enforce compliance and do audits. Why not automate as much of the process and give the development teams access to the same tooling?

Build a tool that checks the list of users authorized in a system against the company HR database to ensure that leavers have had their credentials revoked.

How about a tool that confirms via APIs that all nodes in the cloud infrastructure are built from a secure base image, have been patched within the current patch window, and are placed in security groups appropriately?

These audit and compliance tools help the security team as well as operations to detect mistakes when they happen and ensure that a strong security person can focus her energy and time on actually auditing the really interesting or tough problems.

Setting Secure Baselines

How do you know whether your tools are doing the right job? How can you tell if your product is actually secure?

The real answer is that you can’t ever be certain that your product is really secure, but you can have confidence that your product meets a baseline of security.

By adding security touchpoints into your team’s Agile life cycle, and using your tools and templates, it should be possible to assert what baseline of security you want your product to meet and to be assured that each build coming out of the team meets that level.

This should enable you to have confidence in writing a statement of assurance about your product and to know not only that the build meets that level of assurance, but also that any future builds will continue to meet that level.

What About When You Scale?

The model that we have outlined works well when you have a small number of development teams and a small team of security engineers who can divide the work among themselves equally. This will work for most of the readers of this book. For example, if you have six development teams and two security engineers, then you should be able to scale your security team’s time to handle most of the issues that will come up.

But what if the number of products or services continues to grow?

If you follow Amazon’s two-pizza model,¹ then a development organization of 200 might be made up of 30 or more teams, which means you need at least 10 security engineers to support them. If you follow Netflix’s model of two-person engineering teams, then this model of security won’t scale for you at all. The more development teams you have, the less likely that you can afford to dedicate a security specialist to work with them.

In large organizations, you need to look at application security as a pipeline problem. Instead of trying to solve the problem at the point where it’s no longer possible to cope with the amount of work, your security engineers need to work further up the pipeline, ensuring that development teams are enabled to make security decisions by themselves.

Building Security Teams That Enable

Instead of security teams that do security, you could envision a team that enables security. By this we mean that the team’s primary purpose is to build tools, document techniques, and build capability to develop and deploy secure services. Truly Agile security teams measure themselves on what they can enable to happen, rather than the security issues they have blocked from going out the door.

Creating an environment where the secure thing to do is the easiest thing to do is a great goal for any security team to keep front of mind. It also has the additional positive impact whereby everyone involved is now directly contributing to the security of the development process. This is the only scalable way in which a security team can sustain, if not increase, its impact in a growing organization facing the unfortunate reality that there are never enough funds or qualified practitioners to do all the security work that needs to be done.

Building Tools That People Will Use

By building tools, we mean developing security tooling that can be used by development teams to assure themselves of the security of their products. This might mean looking at the risk management tooling, attack tree analysis, and Agile story tracking tools. Or it might mean automating testing tools to fit into your build pipeline, and automatic dependency inspection tools. Or security libraries or microservices that teams can take advantage of to solve specific problems such as crypto, multifactor authentication, and auditing. It could also be tooling that can safely audit and correct the configurations of your primary security systems, such as your firewalls, or of third-party services, such as those provided by AWS or GCP.

It’s our experience that forcing a team to use a specific tool will produce a compliance or checklist culture, where the tool is an alien artifact that is ill understood and used reluctantly. Development teams should be free to choose appropriate tools based on their needs, and on the risk profile of the system that they are working on. Tools that they understand, that fit into their workflows, and that they will take ownership of.

It’s important to note that these tools should not just identify security defects, but also enable teams to fix problems easily. So a tool that simply does a security scan and dumps the output for someone to review won’t help the average team unless you can link the results to common remediations that are known to work and demonstrate that the tool adds value.

Tools that require significant effort on the part of developers will inevitably end up not being used. Examples of such high user cost are things such as noisy output that requires a developer to actively tease out the parts she cares about from a larger body of superfluous data, or where the actual determination of whether something is important or not requires a developer to take a series of additional and external steps.

When building security tooling, it cannot be stressed enough how important it is to make it easily extensible either through APIs or through the Unix principle of solving a single task and allowing the output to be passed into another tool. View your security toolset as an extensible toolkit that you will continue to add to over time, and where you can combine tools together in complimentary ways without having to rewrite them from scratch every time.

Documenting Security Techniques

There isn’t a great deal written about good application security practices. Security is still viewed as a dark art, practiced only furtively in shadowy corners by learned masters. For many developers, security is mostly about obscure bugs and defensive coding techniques that they think are only needed in special cases.

Security engineers need to teach developers good techniques that are appropriate for your organization. These could cover the steps to safely configure a base web application, usage guidance for working with your cloud service provider in an effective and secure manner, secure coding guidelines and code review checklists for their languages and frameworks, and common risk lists for the kind of application that they are working on.

The key thing is that these techniques need to be applicable, timely, and relevant. NIST guidelines or the UK government’s good practice guide and other common guidance from governments and regulators tend to be so generic and bureaucratic as to be useless to most teams.

Because Agile development teams value working software over documentation, code always trumps paper. Wherever possible, get security guidelines and checklists directly into code: secure headers, secure configuration recipes and playbooks and cloud templates, frameworks with security features enabled by default, and automated security tests and compliance checks that can be plugged into build pipelines and run in production. Code that developers can pick up and use easily, without slowing down.

Key Takeaways

Agile development creates new challenges for security teams. The keys for a successful agile security program are:

Involvement

There are frequent opportunities in an agile life cycle for security and developers to work together, learn from each other, and help each other. Someone playing the role of security on the team (a security engineer assigned to the team or a developer taking on security responsibilities) can be and should be involved in planning sessions, stand-ups, retrospectives, and walkthroughs.

Enablement

Agile teams move fast and are continuously learning and improving, and security needs to help them keep moving and learning and improving instead of blocking them from moving forward.

Automation

Security checks and tests must be automated in ways that they can be easily and transparently plugged into developer workflows and build pipelines.

Agility

Security has to be Agile to keep up with Agile teams. Security has to think and act quickly and iteratively, respond quickly, and keep learning and improving along with developers.

¹ No team should be bigger than you can afford to feed with two pizzas, so between five and seven people, depending on how hungry they are.