I haven’t met too many security professionals that don’t have enough work to do and are looking for more. This may seem obvious to the reader, but in the typical security organization, there is more work to do than there are people to do it. We hear this thought often, but what are some of the reasons behind it?
There are always multiple different factors that contribute to the always-hectic, always-on nature of a security operations and incident response function. Based on own personal experience, along with what I see around the industry, I believe that this intensity comes from two fundamental causes:
● Alert Fatigue: Most organizations simply have too many alerts, too many false positives, and too much noise. This makes it nearly impossible to identify the signal (the true positives) and prioritize work efficiently.
● Lack of Context: Even the best alerting lacks the necessary context to support informed decision making about what the nature of the threat we’re dealing with is and what type of response, if any, is necessary.
Although not everyone in the security community is aware of these fundamental issues, many people are indeed aware of them. We often hear about alert fatigue and lack of context, but what can be done to improve the situation? After all, merely identifying the root causes behind why many organizations find security operations so challenging is not enough. The operational community is more or less aware that these problems exist, but what they’re really after are helpful suggestions, techniques, methodologies, and most of all, solutions.
In the In the past, I’ve recommended that organizations practice spear alerting to reduce alert fatigue and move to a narrative-driven model for security operations to improve context. While I don’t wish to rehash those points here, I do wish to build upon those ideas in this piece.
The astute reader who has read the above pieces and/or has heard me speak on these topics will immediately notice one challenge to building the narrative. Building context around even the highest quality, highest fidelity, lowest noise, reasonable volume alerting requires a fairly diverse set of data to provide that supporting evidence. The ultimate goal of the narrative, and of security operations in general, is to facilitate an informed, educated decision regarding what type of response, if any, is necessary for a given alert or event. In my experience, there are three fundamental challenges that inhibit building the narrative:
● The diversity of data required to build the narrative and the complexity of modern security environment creates a confusing environment in which analysts are unsure of where to go to get the necessary enriching contextual information
● The analytical limitations of many security systems inhibit the ability of analysts to make queries and questions of the data as precise, targeted, and incisive as they need to be
● The performance limitations of many security systems mean even the most precise, targeted, and incisive queries take far longer to return answers than the organization can generally tolerate
This perfect storm of circumstances means that building the narrative manually is a non-starter in almost every environment. But with what options does that leave us? In the past, during my operational career, I have used scripting languages to perform various types of enrichment. Though this was not a fully implemented narrative-driven model for security operations, it was a good start towards that goal. Although automation via scripting can save a tremendous number of analyst cycles, it does have a few issues:
● Scripts require constant maintenance and tuning to remain effective, which takes analyst cycles away from performing analysis and incident response
● Script functionality can be difficult to document, making operational continuity a challenge, particularly when the script author departs the organization
● The percentage of analysts who also have scripting skills is smaller than most people realize, making it even more difficult to recruit the appropriate skill set to the organization
I don’t mean to imply that scripting is never an effective tool for a security organization. Indeed it can be in certain cases. But if we look at the effectiveness and efficiency of a security organization, it is very closely correlated to the quality of its work queue, as that is the primary driver of the organization’s workflow. The effectiveness of that work queue is highly diluted by both alert fatigue and lack of context. For something as central to the effectiveness of a security organization as its work queue, isn’t it time we asked for something better?
It is precisely because of this that I am so excited by the recent emergence of the security orchestration and automation space. This opens up new possibilities for the security operations community that we’ve long been asking for. A few of the benefits I envision this space bringing organizations include:
● The ability to combine and connect multiple alerts through correlation. This, together with spear alerting, has the potential to bring alert volumes way down to far more manageable levels.
● The ability to build the narrative, or parts of the narrative, in an automated fashion. This frees up analyst cycles previously spent on repetitive, manual, time consuming queries for higher level tasks such as analysis, incident response, and hunting.
● Timelier and more informed decision making facilitated by better context.
● Shorter dwell times for intrusions and fewer “missed” intrusions due to reduced noise and improved signal.
● Self-documenting processes for routine, oft-seen alert categories.
● Reduced human error while investigating, documenting, and reporting.
Although the security operations and incident response community is currently weighed down by alert fatigue and a lack of context, I am hopeful for the future. Granted, the extent to which vendors are able to deliver against this set of expectations, as well as the extent to which organizations are able to successfully leverage this capability operationally remains to be seen. Even with this cautionary note, I still see tremendous potential for security orchestration and automation solutions. One thing is for certain — the status quo cannot continue. The alert-driven model for security operations just isn’t working anymore for anyone.
