Incident Response

Foreword

In the beginning, before the first official emergency response team was founded, system administrators routinely handled computer intrusions and security problems. In November 1998, after the Morris Worm, several of us founded the Computer Emergency Response Team/Coordination Center (CERT/CC) in Pittsburgh, PA. Although I departed CERT/CC in 1996, I still have a lot of respect for the organization. While CERT/CC has played an important part in helping to make the Internet more secure, it really was the time and effort of the system administrators who reported incidents and product vulnerabilities to us at the CERT/CC that made the difference. It was not unusual for these dedicated system administrators to spend long days sifting through log files and file systems looking for clues to explain what happened to their systems and to identify other systems on the Internet that had been broken into. Their combined effort is why the words “Coordination Center” are part of the organization’s name.

When CERT/CC began in 1988, security was not on the top of most system administrators’ list of things to worry about. The culture at that time was to have a guest account on your system without a password. If you had any concern about the guest account, you might add a password. The most commonly used password was guest. Of course, everyone knew to try this password. Some of our earliest incidents involved intruders logging into the site’s guest account, exploiting known vulnerabilities, and gaining root access. Because there were so many uninvited guests, this practice eventually became less common. Guest accounts eventually went the way of the dinosaur.

In 1988, information on computer security was hard to find. The resources of Internet were not yet developed and there was very little, if anything, in print to help system administrators deal with security issues. Unless you were part of the “old boys network,” it was very difficult to find any information at all on computer security. CERT/CC changed this. Members of CERT/CC attended conferences, presented papers, gave talks, and held Birds of the Feather (BoF) meetings in an attempt to let system administrators know about the organization and the services it offered.

System administrators began finding CERT/CC and making use of its resources. Although I have no idea of how many incidents I handled directly or worked on with other CERT/CC members; my guess is it’s in the thousands. The most important lesson I learned from all this is that a site should be prepared to handle an incident before it occurs. This saves both time and money, and can help alleviate irreparable damage caused by intruders. Anything a site can do to prepare for an incident will be very beneficial.

One example of a type of Internet incident involved sniffers. An intruder would install a sniffer program on a system to collect login information and passwords for other systems. The intruder would gather the sniffed information periodically and use the information to access other sites or be bartered for new hacks. Sites with proactive computer security policies found the installed sniffers shortly after they had been installed. Prepared system administrators used tools such as tripwire to detect changes in files and directories. These sites were able to remove the sniffer, replace any file altered or replaced by the intruder, and report the problem to CERT/CC, who then notified other system administrators whose sites were affected. Some of these sites had not even been aware that they had been attacked until they received the call from CERT/CC informing them of an intrusion on one or more of their systems!

It was very difficult and time consuming when working with system administrators and others who had no understanding of computer security. Without a computer security policy to follow, the lucky person answering my call often had no idea of whom to call or what to do. Once we worked through that and found all the people who could approve the changes, we were able to stop the sniffer. That was the good news. The bad news is that the system administrator had no idea what had been changed on the system and the extent of the damage. Was the intruder on one system or were other systems involved? Attempting to restore the hacked systems without knowing what was affected could take anywhere from several hours to days. This is when businesses begin to understand the cost of not being prepared for computer security problems.

One of the other important things I’ve learned about incident response is the value of a postmortem. If you ever have an intrusion, when the event is over, gather everyone who worked on the problem for a postmortem meeting. This is a great time to review policies and procedures. It is possible for even the most well-written policies and procedures to age. One incident I worked on early in 1989 involved an intrusion into a computer on a military base. The person answering my telephone call knew exactly what to do. The policy on intrusions was handled by posting armed guards around the facility! In this situation, two guards were dispatched to handle this problem. I have a mental image of two guards with loaded guns attempting to protect a VAX-750 running VMS. Being a Unix fan, I have always visualized the guards shooting the VAX! In reality, no one shot the VAX; unfortunately, they also did not protect it from an intruder. I am sure the policy on intrusions was a wonderful policy when it was created, but it needed to be updated as the times changed!

After all of the years of incident response experience, you might think that the state-of-the-art and the state-of-the-practice would be equal to each other. Sorry to report they are not! “Exploiting Known Vulnerabilities” is a bulleted item in almost every presentation on computer security I have given over the past several years. This problem still exists today! A few weeks before I wrote this foreword, I handled an incident. It seemed like old times. The owners of a web server discovered their software would not run on the latest version of RedHat Linux. Instead of correcting their software, they found it took less time to load an older version of RedHat Linux. It only took two weeks for an intruder to gain access to their system and trash it. The server was down for days while they reloaded the server and attempted to correct all of the known security problems. This server hosted several customer web sites so, while it was down, the web hosts customers lost revenue through their web sites.

Another incident occurred last summer. A company asked if I would help them with an email problem. They believed confidential email messages were finding their way to a competitor. Well, “Exploiting Weak Passwords” is right up there with “Exploiting Known Vulnerabilities.” This company used email accounts with passwords that matched the account name. For example, the dehart account had dehart for the password. While this made it easy for their users to remember their passwords, it made it all too easy for an intruder to. . . never mind; you know where I am going with this. For many sites their state-of-the-practice is no better today than it was in 1988.

Preparing to handle a computer security incident can be a difficult challenge. It is definitely worth the time. Today, there are several incident response teams. Each makes information available for you to use. There are several sources of information, for example, mailing lists. In addition to all of the great information provided on incident response in this book, the list and description of system administration tools in this book is priceless. These tools help system administrators keep their systems secure. Many of the tools are so good that the intruders use them, too! Take the time to download and become familiar with them. Use these tools to learn what the intruders may already know about your site.

— Ed DeHart