Most of the complex fields humans engage in develop their own terminology, which then becomes a problem of translation for the expert. One of the fundamental problems for security at the “C-level” is to translate between security’s inner language, which tends to be about risk, and business’ inner language, which is about money and opportunity. The problem a lot of us security practitioners have is that we can’t really talk sensibly about risk without trying to quantify it, because the others at the “C-level” are going to want to make a risk/reward judgment and, all too often, we’re kind of waving our hands and the best we can do is point to some of the other casualties by the road-side and say “well, look what happened to them!”
Metrics are critical for our ability to do better at our jobs, to quantify where we are succeeding and to measure what works, and how well it’s working.
A great deal of the responsibility for our difficulty with metrics is because of the rapidly changing technological landscape that we’re dealing with. We had a decade or so to get used to firewalls and anti-virus and then the WWW came along, and a decade after that it’s “Cloud Computing” and we can’t really be said to understand the benefits of a firewall, yet. In the meanwhile, the firewall we’re talking about has changed profoundly, as well. Unlike in fields where there are large-scale demographics and metrics, we’re stymied by our inability to make truly effective recommendations by the lack of underlying information about the long-term effectiveness of what we’re recommending. For example, automotive safety studies can tell us (with some confidence) that you’re reducing your chance of injury by 90% if you wear your seatbelt. Have you ever been asked “do we really need a firewall?” and been left with your jaw hanging and your hands waving?
Senior security practitioners (here, I’m talking about the people who report to the CSO/CISO) are working at the level where metrics are important. It’s not the case that a security practitioner is going to have to walk into a meeting with a CEO or board of directors, and make a case for something-or-other with a bunch of PowerPoint pie charts. That’s the wrong level of the discussion for metrics. The metrics come in where the senior security practitioners develop understanding about what’s going on in their organization, and provide that information to the “C-level” as input into critical decisions. I’m sure that there have been cases where security metrics wind up being presented to the CEO, but, in general, we’re talking about back-and-forth at the higher end of the technical contributors and between business units.
Before we go further, let me offer a couple of quick definitions for security metrics. A fairly formal way of putting it would be: Security metrics are data points that provide analytical insights into an enterprise security posture.
Metrics provide insights into the dimensions of an enterprise cyber security program and usually consist of things like: counts of things – numbers of servers, numbers of incidents, numbers of vulnerable hosts; risk extrapolations based on enterprise’s assessments of importance and vulnerability, operational effectiveness “speeds and feeds” or comparative benchmarks that measure how security practice compares with other organizations in the same space or scope.
My favorite definition of metrics is: A metric is some data and an algorithm for reducing and presenting it to tell a story.
Ultimately, the metrics exist to justify your existence at that level of the organization. If you can’t show that you’re succeeding or failing, someone else is going to leap to their own conclusions on that matter – which means you’d better hope they convince themselves you’re effective, rather than not. In fact, I’d argue that if you focus your metrics production on that one question: “Am I succeeding?” you’re probably heading in the right direction, metrics-wise. The answer might be “no” in which case you’re still heading in the right direction, metrics-wise though your career may be in a nose-dive otherwise.
As a set of corollaries to that, there are some more detailed purposes for metrics:
• To justify your budget
• To argue for change
• Because it’s fun
I read somewhere that “Math is the language of physics” and I’m tempted to say that “Metrics are the language of security” but doing that would reveal the sorry state that security is, in a lot of places. We are only just starting to be able to collect more useful information about breaches (see my interview with Jay Jacobs here) but we’re still a far cry from being able to make prescriptive declarations like: “You should wear your seatbelt and here’s why”)
Source: National Highway Transportation Safety Administration
And, make no mistake, that kind of declarative prescription is what security practitioners at the senior and C-level are here to do. The chart above is an example of what I’d consider a well-presented metric; it’s the kind of thing that makes its case simply and quickly, and focuses the reader’s mind directly on the problem it is analyzing. If the question we were trying to ask was “Do seatbelts work?” the only argument that chart would leave is debaters’ arguments about “what does ‘work’ mean?” or questions about injuries versus fatalities, etc. During the remainder of this series we’ll get into those kinds of questions in more detail. For now, we should be thinking about how to make a case for security that is that effectively communicated.
Next up: We’ll dig into The Scientific Method and how metrics are a critical component of mankind’s greatest and most important activity, and the relationship between metrics and statistics.
