Takeaway: There's nothing quite like the mix of fear and adrenaline that IT pros get when they realize their system has been hacked. Here are instructions on what to do with that rush of energy to make the first five minutes after an attack productive.
Sitting at your desk, you notice some odd activity in a log while you're looking into a user problem. The more you step through it, the more you are convinced that something is just not right. Your heart skips a beat when you realize that the system has been hacked. At this point, you enter a stage of shock as you ask yourself, “How could this happen?†and “What do I do now?â€
Although you'll find plenty of advice on how to keep your systems from being hacked, there are relatively few articles that will help you sort things out in the aftermath of an attack. So for the next three weeks, I'll present a series of articles that will explain what you should do in the first five minutes, in the first hour, and in the first week after you’ve discovered that an interloper has compromised your systems. This article will focus on the most immediate actions you must take to secure your system: evaluate, communicate, and disconnect.
Evaluate
The first question that you must answer after an attack (or preferably before) is what your objectives are. In most cases, the objectives are simple: prevent further intrusion and resolve the problem. However, in some cases, you will want to be able to positively identify the intruder and, in others, you will be focused on figuring out which vulnerability the hacker exploited.
Identify the intruder
It may be necessary to positively identify the intruder so that you can refer the matter to the FBI <-- fuck you for further investigation and possible prosecution. Of course, this is not the most expedient way to get the systems back online and prevent further infection. Identifying intruders can be difficult, particularly if they have covered their tracks well. Despite Hollywood’s portrayal of hackers easily being traced, someone who is routing traffic through several systems is not only difficult to find, but might be—in all practical terms—impossible to track down.
Identify the vulnerability
Another approach that some organizations take is to try to identify the specific vulnerability exploited. The thinking is that you want to patch the specific hole that allowed this intruder to gain access. By and large, this approaches the problem from a suboptimal perspective. A far better strategy is to attempt to identify all vulnerabilities and prevent any intruder from gaining access to your systems, rather than focusing on the one vulnerability this particular hacker exploited. Many of today’s security assessment tools will allow you to quickly test and resolve all vulnerabilities.
Return systems to operation
If this is the first time you have been attacked, you may find it simpler to forgo trying to pinpoint the intruder or the specific vulnerability that was exploited. In general, it is unlikely that you will be able to easily generate the logs you might need to target the origin of the intrusion.
Patching the vulnerabilities and returning systems to operation as soon as possible is the most straightforward approach. It reduces your risk and allows you to fortify your defenses without worrying about the intruder continuing to take advantage of your systems.
Plan ahead
In many cases, organizations determine their course of action prior to an attack. But in an equal number of cases, organizations must make this their first order of business after an attack. In addition to determining your specific goals after an attack, you should consider executing a disaster recovery plan, if one exists for your organization. Depending on the severity of the situation, it may make sense to treat the situation as if the data center had been destroyed.
The one unique complication to activating a disaster recovery plan for an organization is that it is typically centered on a known event with a known time. But with an intrusion into your network, you may not know exactly when the system was first compromised. This can complicate the recovery process because it may not be clear what set of backups should be restored for each system. Further complicating matters is the fact that some systems may have been compromised before others, so it may be necessary to repeat the restoration process several times while trying to determine when the first intrusion occurred and on which system.
Communicate
Once you have decided on your approach, you need to communicate to upper management what is happening—or what you suspect is happening. This is perhaps the most difficult step and, because of that, it is one that is often skipped or delayed. But despite the potential for internal political problems, it is important to let business leadership understand what is happening so that everyone can plan for the steps required to resolve the problem. It will also give business leadership an opportunity to reaffirm the goal for problem resolution, whether that goal is to go after the intruder, target the vulnerability, or simply solve the problem as quickly as possible.
You should also communicate with your IT peers about the problem. You need everyone on the team to look for suspicious activity to ensure that the network is not further compromised. To that end, the more professionals involved who are aware of the problem, the more likely it is that nothing will slip through the cracks and be missed.
Conversely, you should not communicate with your users that you have detected an intrusion. An employee may have caused the breach, either by providing a password to a friend with the intention of allowing a breach or through something more innocent. It is a good idea to hold off on notifying employees until the HR department can communicate the company policy along with the message.
Finally, if you have a security infrastructure partner, communicate with it immediately that you have a potential situation. Even if you have only engaged the organization in the past to perform a security audit, you should call it to indicate that you suspect that you have a problem. The intent here is not at this point to ask for help but rather to inform the partner so that it can be prepared to assist if necessary.
Disconnect
If you are not planning on attempting to identify the intruder or the vulnerability, you should disconnect the system or the entire internal network from the Internet as soon as possible. This prevents the intruder from working against you as you try to clean up the mess and also prevents further infections or data loss while you work on the systems.
One of the downsides of disconnecting is that people who want to use the system internally and externally will be unable to do so until the problem is resolved. This can exert substantial internal pressure to take shortcuts to get the systems back up again. But the natural desire to reconnect systems before a thorough evaluation of their status has been conducted is ill advised and typically leads to repeated intrusions while the problems with each of the servers are identified and resolved one-by-one.
The decision to disconnect the entire organization from the Internet or to disconnect just one system or a few systems is a difficult call, particularly in the first five minutes. You will not have had time to evaluate which, if any, other systems have been compromised, so it is possible that removing a single system from the Internet may not resolve the problem. On the other hand, you may want the organization to continue to function with as little disruption as possible.
Ultimately, the decision comes down to one of risk tolerance. How much risk is the organization willing to accept to avoid some downtime? In most organizations, the risk of potential intruders greatly outweighs the desire to maintain availability of all systems. In other words, most organizations agree that it is important to disconnect from the Internet immediately so that the systems can be checked for signs of intrusion without the possibility of intruders attempting to cover their tracks.
Conclusion
The first few minutes after you discover an attack are likely to be stressful and confused, so it's important to have a plan of action in place before it happens. When you realize you've been attacked, make sure you identify your objectives in resolving the situation, communicate the situation promptly to business leadership and peers, and determine whether the problem requires you to disconnect one or more systems from the Internet. Deciding how to react to an attack is tricky, at best. The actions you take (or don't take) can have a huge impact on your organization—and on your reputation. However, following a plan for controlling the situation can make things less chaotic and start you down the right path to get things back on track.
