O'Reilly logo

The Shellcoder's Handbook: Discovering and Exploiting Security Holes, Second Edition by Gerardo Richarte, Felix FX Lindner, John Heasman, Chris Anley

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 14. Protection Mechanisms

With the rise of code execution bugs and exploitation, operating system vendors started adding general protection mechanisms to protect their users. All of these mechanisms (with the exception of code auditing) try to reduce the possibility of a successful exploit but don't make the vulnerability disappear, hence, every minor glitch in the protection mechanisms could be leveraged to re-gain code execution.

This chapter first describes the generalities of the most common mechanisms, and then talks about the specifics of each operating system. It also presents some weaknesses in the protections and what would be needed to bypass them.

Protections

Throughout the chapter there is the need to show different stack layouts as different protections are applied. The following is the C code used as an example:

#include <stdio.h>
#include <string.h>

int function(char *arg) {
   int var1;
   char buf[80];
   int var2;
printf("arg:%p var1:%x var2:%x buf:%x\n", &var1, &var2, buf);
   strcpy(buf, arg);
}

int main(int c, char **v) {
  function(v[1]);
}

The standard stack layout, without any protections or optimizations, looks like the following.

↑ Lower addresses

var2

4 bytes

buf

80 bytes

var1

4 bytes

saved ebp

4 bytes

return address

4 bytes

arg

4 bytes

↓ Higher addresses

Note that the chapter uses the same convention as every known debugger: lower addresses are shown higher up the page.

Non-Executable Stack

A very common class of security bugs, even today, is the stack-based buffer overflow, and ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required