At the recent FIreEye Cyber Defense Summit in Washington, DC, I had the privilege of hearing General (ret.) Colin Powell deliver one of the keynotes. He is a wonderful speaker, and spoke about many important topics, principles, and ideas. As it turns out, one of his quotes in particular had a profound impact on and resonated very strongly with me. During the course of his keynote, he stated: “The most important thing about a decision is that it’s timely”.
I found this quote to be quite insightful. He was speaking to us from his experience in military and diplomatic leadership positions, but this basic tenet holds true for information security as well. Security professionals need to be able to make better, more informed decisions, and they need to be able to make them in a timely manner. In my estimation, this is one of the biggest challenges we face in the security field. I’d like to spend the rest of the piece examining why I believe that is the case, as well as how I believe we can address the issue as a profession.
There is no shortage of people who parade around talking about the concepts of excessive dwell time (205 days) and excessive time to remediate (32 days). The question I always ask is: Why? Why is this the case? Why are we in this situation? Why do most organizations still struggle with this?
Although there are many approaches one could take to answer this question, I’d like to take what may initially seem to be a radical approach. I think that as I build out my argument though, my approach will seem far less radical and far more intuitive. I base my observations upon my operational experience, along with my interactions and discussions with security professionals currently in operational roles. I would say that we find ourselves in our current state of affairs largely because we can’t make decisions in a timely manner. This is precisely what General Colin Powell was referring to during his keynote address.
Now, to clarify, I don’t mean to imply that we as security professionals are indecisive. Rather, what I’m saying is that our ability to make informed decisions is severely hampered, primarily by two factors:
● Low signal-to-noise ratio: Simply put, most organizations suffer from severe alert fatigue. We are drowning in information, but starved for knowledge. Data is everywhere, but our ability to draw meaningful conclusions from it is painfully limited. How do I know if alert #12,345 of the 100,000 alerts in my daily work queue is a false positive, or the alert that will land me in the newspaper in six months’ time?
● Lack of context: Decisions cannot be made in a vacuum. Alerts contain a snapshot of a moment in time. A more complete picture of what happened before, during, and after an alert is necessary in order to make an informed decision. Unfortunately, the context crucial to supporting the decision making process often proves elusive to most organizations. Getting that supporting context is typically a manual process that requires far too many human resources and time. That impedes decision making.
Making an informed decision in a timely manner requires a more complete picture, or story of what occurred. I often call this a narrative. That’s all well and good, but how can organizations replace their noisy, voluminous queue of context-less alerts with a high fidelity contextual queue of narratives?
While moving to a queue of narratives is not an easy process, it’s one that begins with re-examining the incident response process in an effort to improve it. Although the incident response process can be represented in a few different ways, I would like to abstract it to three high level functions: Detection, Analysis, and Response.
When we look to improve this process, we ought to begin at the beginning, or more appropriately, one step before the beginning.
Reducing false positives and raising the signal-to-noise ratio begins with better detection. And better detection begins with better content feeding the detection process. I, along with many others, call this content development. The goal of content development should be to alert on all of the activity that matches the risks and threats of concern to the organization, while alerting on none of the activity that does not match those risks and threats. Although the length of this piece does not permit a more detailed discussion of this topic, it is something I have written about in the past. A focus on content development produces a much more streamlined and lower-noise queue of alerts from which we can begin to build our narratives.
Even a reliable, high fidelity alert, though, is just a snapshot of a moment in time. It, in and of itself, does not contain enough context to facilitate informed decision making. What the alert needs is enrichment from network, endpoint, mobile, and intelligence sources to put the pieces of the puzzle together. This analysis is the core of building the narrative. In my experience, most organizations build the narrative manually, even though 80% of the queries used to do so are the same all the time. Automation, whenever possible, is key here, and where automation ends, the analyst’s duties begin. The goal here is simple: reduce the time required for the analyst to interleave all of the necessary supporting evidence and reach a conclusion. That is what facilitates an informed decision, and the timelier, the better. Why? That’s where a discussion of response can shed some light.
Response is about taking action on those informed decisions. Once the threat is understood, it should be contained and remediated. The goal should be to respond quickly enough so as to minimize or eliminate any damage to the organization. Inability to draw a timely conclusion regarding what has occurred is the main reason that organizations struggle to adequately contain and remediate threats. The result is that these threats persist, and in the end, the organization often suffers great damage because of it.
While improving the security posture of an organization involves many details, it begins with a focus on timely decision-making. There are many supporting elements that facilitate timely decision-making, of course. But without this focus, an organization cannot expect to improve its security posture. As General Powell stated, the most important thing about a decision is that it’s timely. We ought to do everything in our power to make timely decisions a reality.