Warning: This section is going to be a bit “meta-” and it can’t be helped. You can’t talk about abstract meta-analysis any other way!
Don’t just ask your boss, “what metrics should I collect?” Metrics are ‘produced’ not ‘collected’; you collect underlying data to produce the metrics, so if you ask that question, you’ve just shown that you don’t understand metrics. More importantly, you’ve just shown that you don’t understand what you do. Rather than having an existential crisis, you need to spend some time figuring out what metrics are appropriate for your organization, which really means figuring out your organization’s purpose or product.
Metrics are going to be very different for various parts of an organization. They have to be, because otherwise organizations would be monolithic. If there is some kind of grouping of purpose in an organization, that grouping is going to provide your first principles for figuring out what metrics to keep. If you are in the Information Technology (IT) department, your metrics will be related to IT matters. If you’re in Human Resources, your metrics are going to be related to HR subjects, and so forth. The place where you would start thinking about monolithic metrics across organizational sub-divisions is if you are an executive trying to do “business intelligence” analysis of how the sub-divisions work with one another.
Generally, your metrics are going to be outcome focused or operations focused, such as how many widgets you produced, or how you produced them. IT security is an interesting problem because security metrics tend to focus on the difference between what is working right now, and what is likely to break. We’re not dealing with a simple “materials in, widgets out” type of analysis, because we immediately fall into the problem IT security faces; we’re concerned with an unpredictable future. IT security metrics are going to tend to be oriented toward figuring out efficiency, effectiveness and long-term outcomes. That’s why it’s hard!
It’s important to realize there is no “one size fits all metric” because there is no one size fits all organization. There is plenty of room to keep common data-points in our community, such as “number of email messages sent” or “number of users with administrative permissions,” but those are merely interesting details. What we’re trying to do is the hard part – exploring the connection between what security does and what the business does, and trying to quantify it.
What are some of the things security does? Here are a few possibilities:
• Protecting customer data
• Protecting our company’s intellectual property
• Managing reputational risk
• Preventing service interruptions
• Responding to incidents
• Deflecting denial of service attacks
• Managing vulnerabilities and configurations
• Assessing, analyzing, and recommending technologies
If you look at the list above, and you see a couple of the items as being ones that IT security “owns” in your organization, you can then ask yourself about the common elements between the items you’re responsible for. Then, start to map them mentally into the way your security department goes about doing them. For example, preventing service interruptions, incident response, and dealing with denial of service attacks are likely to overlap functionally – you may want to analyze that capability as a single unitary function. Technology strategy and the highly nebulous goal of “protecting customer data” may be a meta-function of supporting what amounts to a consultative practice for executives and business units.
Don’t worry, we’re not going to start slapping measurements on something as vague-sounding as “protecting customer data” – yet – but since it’s part of what you may be responsible for, we need a framework for figuring out what can be measured and how to make it relevant. How do we do that? To start, we need to do a deeper analysis of what you do when you “protect customer data.” What does the process of protecting customer data look like, the way you do it? What sub-steps in that process require which resources and in what quantity? Is that process repeatable enough that it’s somewhat predictable?
In theory, once someone begins this kind of analysis, they may discover that their organization is completely random, and their security process is therefore more or less nonexistent. That’s the theory, but in practice I suspect that anything other than a start-up in its first couple months of operation will already have some kind of established processes that can be analyzed. If you’re reading this and you’re thinking, “we have no processes at all, everything we do is always chaos!” then you may be one of the rare people who works for an organization that isn’t organized at all.
Next up: Establishing your own metrics: a framework for figuring out what to do.