Essential System Administration, 3rd Edition

Prelude: What’s Wrong with This Picture?

Before turning to the specifics of securing and monitoring Unix systems, let’s take a brief look at three well-known historical Unix security problems (all of them were fixed years ago):

• The Sendmail package used to include a debug mode designed to allow a system administrator to type in raw commands by hand and observe the effects. Unfortunately, because anyone can run the sendmail program, and because it runs as setuid root, a nefarious user could use sendmail to execute commands as root. This is an example of a security hole created by a back door in a program: an execution mode that bypasses the program’s usual security mechanisms.

• Traditionally, the passwd -f command enabled users to change the information in the GECOS field of their password-file entries. However, as originally implemented, the command simply added the new information to the user’s GECOS field without examining it first for characters such as, for example, colons and new lines. This oversight meant that a treacherous user could use the command to add an entry to the password file. This is an example of a program’s failure to validate its input. The program simply assumes that the input it receives is valid and harmless without checking that it is in the form and length that is expected.

  Another variation of this problem is called a buffer overflow . A buffer overflow occurs when a program receives more input than the maximum amount that it is able to handle. When it later ...