Security Experts:

Connect with us

Hi, what are you looking for?


Incident Response

Establishing Your Own Metrics: What Not to Do

Warning: This section is going to be a bit “meta-” and it can’t be helped. You can’t talk about abstract meta-analysis any other way!

Warning: This section is going to be a bit “meta-” and it can’t be helped. You can’t talk about abstract meta-analysis any other way!

Don’t just ask your boss, “what metrics should I collect?” Metrics are ‘produced’ not ‘collected’; you collect underlying data to produce the metrics, so if you ask that question, you’ve just shown that you don’t understand metrics. More importantly, you’ve just shown that you don’t understand what you do. Rather than having an existential crisis, you need to spend some time figuring out what metrics are appropriate for your organization, which really means figuring out your organization’s purpose or product.

Metrics are going to be very different for various parts of an organization. They have to be, because otherwise organizations would be monolithic. If there is some kind of grouping of purpose in an organization, that grouping is going to provide your first principles for figuring out what metrics to keep. If you are in the Information Technology (IT) department, your metrics will be related to IT matters. If you’re in Human Resources, your metrics are going to be related to HR subjects, and so forth. The place where you would start thinking about monolithic metrics across organizational sub-divisions is if you are an executive trying to do “business intelligence” analysis of how the sub-divisions work with one another.

Generally, your metrics are going to be outcome focused or operations focused, such as how many widgets you produced, or how you produced them. IT security is an interesting problem because security metrics tend to focus on the difference between what is working right now, and what is likely to break. We’re not dealing with a simple “materials in, widgets out” type of analysis, because we immediately fall into the problem IT security faces; we’re concerned with an unpredictable future. IT security metrics are going to tend to be oriented toward figuring out efficiency, effectiveness and long-term outcomes. That’s why it’s hard!

It’s important to realize there is no “one size fits all metric” because there is no one size fits all organization. There is plenty of room to keep common data-points in our community, such as “number of email messages sent” or “number of users with administrative permissions,” but those are merely interesting details. What we’re trying to do is the hard part – exploring the connection between what security does and what the business does, and trying to quantify it.

What are some of the things security does? Here are a few possibilities:

• Protecting customer data

• Protecting our company’s intellectual property

• Managing reputational risk

• Preventing service interruptions

• Responding to incidents

• Deflecting denial of service attacks

• Managing vulnerabilities and configurations

• Assessing, analyzing, and recommending technologies

If you look at the list above, and you see a couple of the items as being ones that IT security “owns” in your organization, you can then ask yourself about the common elements between the items you’re responsible for. Then, start to map them mentally into the way your security department goes about doing them. For example, preventing service interruptions, incident response, and dealing with denial of service attacks are likely to overlap functionally – you may want to analyze that capability as a single unitary function. Technology strategy and the highly nebulous goal of “protecting customer data” may be a meta-function of supporting what amounts to a consultative practice for executives and business units.

Don’t worry, we’re not going to start slapping measurements on something as vague-sounding as “protecting customer data” – yet – but since it’s part of what you may be responsible for, we need a framework for figuring out what can be measured and how to make it relevant. How do we do that? To start, we need to do a deeper analysis of what you do when you “protect customer data.” What does the process of protecting customer data look like, the way you do it? What sub-steps in that process require which resources and in what quantity? Is that process repeatable enough that it’s somewhat predictable?

In theory, once someone begins this kind of analysis, they may discover that their organization is completely random, and their security process is therefore more or less nonexistent. That’s the theory, but in practice I suspect that anything other than a start-up in its first couple months of operation will already have some kind of established processes that can be analyzed. If you’re reading this and you’re thinking, “we have no processes at all, everything we do is always chaos!” then you may be one of the rare people who works for an organization that isn’t organized at all.

Next up: Establishing your own metrics: a framework for figuring out what to do.

Written By

Click to comment

Expert Insights

Related Content

Application Security

Cycode, a startup that provides solutions for protecting software source code, emerged from stealth mode on Tuesday with $4.6 million in seed funding.

Data Breaches

GoTo said an unidentified threat actor stole encrypted backups and an encryption key for a portion of that data during a 2022 breach.

CISO Strategy

Cybersecurity-related risk is a top concern, so boards need to know they have the proper oversight in place. Even as first-timers, successful CISOs make...

Risk Management

A threat-based approach to security often focuses on a checklist to meet industry requirements but overlooked the key component of security: reducing risk.

Incident Response

Cygnvs emerges from stealth mode with an incident response platform and $55 million in Series A funding.

Risk Management

CISA has published a report detailing the cybersecurity risks to the K-12 education system and recommendations on how to secure it.


A recently disclosed vBulletin vulnerability, which had a zero-day status for roughly two days last week, was exploited in a hacker attack targeting the...

Data Breaches

T-Mobile disclosed another massive data breach affecting approximately 37 million customer accounts.