Computer Security Incident Response Flowchart
This document outlines the basic steps to be taken in the event of a computer security incident.
Because of the wide range of possibilities, this document will be restricted to generic approaches for Unix systems.
This document has the following outline:
Determining if there has been an incident
Preparing for an investigation
Collecting and archiving evidence
Undeleting files
Establishing trust
Things to look for
Resolving incidents
In most cases, you will already have a good idea if there has been an incident.
You may not know, however, the extent of the incident.
A machine may exhibit suspicious behavior due to one or more of the following:
- hardware problems
- software problems
- accidental deletion of system or user files
- malicious user
- hacked user account
- hacked root account
- denial of service (DoS) attack
Before you do anything you need to decide what you are hoping to accomplish.
- In most cases, the goal will simply be to restore service and close the hole that allowed the incident.
To accomplish that goal, you need to:
- determine what allowed the incident to occur (what vulnerability was exploited?)
- determine the extent of the incident (did they get root, or just a user account?)
- If there is not a known vulnerability, it is important to find the vulnerability to prevent a repeat attack.
- If it might be necessary to seek disciplinary action against the intruder, then it is important to follow rigorous steps as outlined in the section on Collecting and archiving evidence.
It is generally best to collect/archive everything, but practical aspects may need to be considered.
A guideline has been given in RFC 3227.
For a formal investigation, look at The Coroner's Toolkit, available at www.fish.com/tct/ or at www.porcupine.org/forensics/tct.html.
In the event that you're actually planning to take this to court, there are some important things to keep in mind:
- Log everything; log often: This means you should take notes with pen and paper, noting the time, what you're doing, why you're doing it, and what you've found. It may be a good idea to sign/date each page.
- Maintain a chain of evidence: In court you will need to testify that there is no way that any system files could have been tampered with during the investigation. This means you'll need to keep a copy of the original data somewhere safe. Work from a copy (use dd), not the original.
- Get the complete picture: using a tool like TCT can help.
- Take advantage of redundancy: don't trust any single source of information. Try to have multiple witnesses of any event.
Depending on the specifics of your filesystems, it may be possible to undelete files.
On older DOS/Windows systems, utilities like undelete could be used to recover accidentally-deleted files.
Newer Windows systems utilize a "Recycle Bin" to help prevent accidental deletion.
Unix systems, however, generally have filesystems optimized for performance, and therefore file undeletion can be extremely difficult or impossible.
TCT contains a utility to undelete files from non-journaled filesystems.
Be warned that it requires 220% of the unused space on the filesystem.
This means if you're using 8GB on a 10GB partition, you will need 4.4GB of free space on a separate partition to recover your files.
Also, there's no guarantee your information is still there.
Even if it is, you're on your own for finding it among that unused 2GB of random data!
Nevertheless, if it's an extremely critical file, and you know keywords that it contained, and you're willing to wait several hours, then this might be worth trying.
If you're trying to recover files from an ext2 filesystem, you might also want to check out the information available at www.praeclarus.demon.co.uk/tech/e2-undel/.
When investigating a possibly-hacked machine, it is important to recognize that not all system utilities can be trusted.
It is common for rootkits to install trojaned binaries.
For example, t0rnkit8+linux installs trojaned binaries for
/bin/ps /sbin/ifconfig /bin/netstat /usr/bin/top /usr/bin/slocate /bin/ls /usr/bin/find /usr/bin/dir /usr/sbin/lsof /usr/bin/md5sum /sbin/syslogd /usr/bin/pstree.
Note that md5sum is in that list -- the new version reports the old sums for these files!
Similarly, t0rn matches the length of the files to the previous length (so an `ls -l` won't show a filesize difference) and matches the modify and access times to the previous values.
On a system I recently investigated, I noticed that t0rn did not update the change time on modified files, so an `ls -lc` was useful for tracking down changed files.
It is also possible, though more difficult, to trojan various system libraries.
The result is that, at minimum, you should investigate your system using a trusted set (ie., from a CD) of statically-compiled binaries.
When trying to determine how, or if a machine was hacked, there is a lot of information to sift through.
This section will give a list of things to remember to check for.
- log files: system log, httpd log, last log, su log, ftp logs
- open ports: netstat -an | grep LISTEN or run nmap against the machine
- suid binaries: find / \( -perm -4000 -o -perm -2000 \) -print
- modified files: tripwire, ls -lc, md5sum, find / -name ..\?\*
- mysterious processes
- crontabs (both root and user) and at jobs
- patch history: attackers will sometimes patch a machine so it will not appear to have been vulnerable to compromise
Note that system crackers often hide things in plain sight.
For example, t0rn adds the text file /lib/lidps1.so.
In a recent incident, a hacked machine was patched, correcting the vulnerability by which it had been hacked.
A sysadmin might be tempted to believe that the hacker was trying to be "helpful" by patching her machines.
A careful analysis showed, however, that that hacker had also installed ssh listening on a high-numbered port to give themselves future root access.
Look over the goals you set when preparing for the investigation.
You should, at minimum, have achieved your goals.
A rough guide of what you will need to have done is:
- no incident: document what led to the false alarm
- malicious user: document what the user did in order to establish a pattern of behavior
- hacked user account: change user's password, educate the user on selecting good passwords and secure access methods
- hacked root account: reinstall system and patch vulnerabilities
In addition, you should inform anyone affected of the outcome of your investigation.
This may include security@, affected users, and other organizations.
In the event of an unknown vulnerability being compromised, it would also be good to report it to bugtraq or to incidents.
MAC: Modification, Access, Change
touch -m updates the modify time
touch -a updates the access time
can't (easily) update the change time?
Damian Menscher: menscher@uiuc.edu