I’ve written in the past on the topic of “alert fatigue”, which describes the concept where organizations receive too many alerts with too little context and too many false positives. I discussed this concept most recently in a SecurityWeek piece entitled, “Security Operations: What is Your Signal-To-Noise Ratio?”
I received some great feedback on that piece. The feedback I received reaffirms my belief that security professionals know the pain of alert fatigue and the deluge of false positives all too well. Not surprisingly, many people also asked me how they can raise their signal-to-noise ratio. That is an excellent question for which I am happy to offer some advice.
I have found it most effective to first enumerate security risks, goals, and priorities as discussed in one of my previous SecurityWeek pieces entitled, “Is Security An Unsolvable Problem?” Following that, I advise organizations to then “Throw Out The Default Rule Set”, as I discussed in another one of my previous SecurityWeek pieces.
This approach is a bit different than the traditional approach taken by many security organizations, and at first, it may seem radical. But we already know that the conventional approach drowns us is noise and obscures our signal to the point that we cannot operate. It would seem to me that a different approach is necessary, one similar to that which I have described above. I call this approach “spear alerting”, and it is the only way to allow an organization to get to 100 alerts a day. What do I mean by that? Allow me to explain.
One of the goals of an incident response team should be to handle no more than 100 alerts a day. At first, this may sound like a ridiculous assertion. However, I think that if we examine this more closely, you will agree that it makes sense. Let’s take an analytical approach and go to the numbers.
Many organizations have a stated goal of moving from detection to containment in one hour. Put another way, one hour should be the time allotted to work an alert, perform all required analysis, forensics, and investigation, and take any necessary containment actions.
Let’s say we have each of our analysts working an eight hour shift. Assuming 100% productivity for each analyst, that allows each analyst to work approximately eight incidents per day. Let’s assume that we want to work 96 alerts properly each day (since 100 is not divisible by eight). That works out to a requirement to have 12 analysts on shift (or spread across multiple shifts) to give proper attention to each alert.
What happens if analyst cycles are taken away from incident response and lent to other tasks as sometimes happens? The numbers look even worse. What happens if the necessary analysis, forensics, and investigation take more than an hour (due to technology, process, or other limitations)? The numbers look even worse yet.
So, if you’re the type of enterprise that has 500 analysts sitting in your SOC or Incident Response Center, you can probably stop reading this blog post and get back to your daily routine. What’s that you say? The analyst is the scarcest resource, and you don’t have enough of them? Yes, of course. I know.
Let’s face it — the numbers are sobering. Even a large enterprise with a large incident response team can realistically handle no more than 100-200 alerts in a given day. Sometimes I meet people who tell me “we handle 5,000 incidents per day”. I don’t believe that for a second (putting aside, for now, the fact that incidents, events, and alerts are not the same thing). Either that organization is not paying each alert the attention it deserves, or the alerts are of such low value to security operations that it wouldn’t make much difference whether they fired or not.
As we can see, the challenge quickly becomes populating the work queue with fewer more reliable, higher fidelity, more actionable alerts for analysts to review. This process is what I’m referring to as spear alerting and can be outlined at a high level as follows:
• Collect the smallest amount of data of highest value and relevance to security operations and incident response that provides the required visibility.
• Identify goals and priorities for detection and alerting in line with business needs, security needs, management/executive priorities, risk/exposure, and the threat landscape. Use cases can be particularly helpful here.
• Craft human language logic designed to extract only the events relevant to the goals and priorities identified in the previous step.
• Convert the human language logic into precise, incisive, targeted queries designed to surgically extract reliable, high fidelity, actionable alerts with few to no false positives
• Continually iterate through this process, identifying new goals and priorities, developing new content, and adjusting existing content based on feedback obtained through the incident response process.
In resource-limited environments, every alert counts. Since most of us work in such environments, we need to ensure that we populate the work queue with only reliable, high fidelity, actionable alerts. Fans of the conventional approach may say, “If I reduce from 100,000 alerts a day to 100 alerts a day, I may miss something.” To those people, I would ask the following question: If you never look at 99% of your alerts, or you quickly dismiss them as false positives, what is the point of firing those alerts and what value do they add to security operations? Further, are you certain that you would not miss important alerts because their signal would be lost in the noise?
Alert fatigue is a serious problem in the security operations and incident response world today. Spear alerting is an approach that can help organizations improve their signal-to-noise ratio and make their security programs much more efficient and effective. This goes a long way towards improving the overall security posture of the organization.