Connect with us

Hi, what are you looking for?


Incident Response

Establishing Your Own Metrics: What Not to Do

Warning: This section is going to be a bit “meta-” and it can’t be helped. You can’t talk about abstract meta-analysis any other way!

Warning: This section is going to be a bit “meta-” and it can’t be helped. You can’t talk about abstract meta-analysis any other way!

Don’t just ask your boss, “what metrics should I collect?” Metrics are ‘produced’ not ‘collected’; you collect underlying data to produce the metrics, so if you ask that question, you’ve just shown that you don’t understand metrics. More importantly, you’ve just shown that you don’t understand what you do. Rather than having an existential crisis, you need to spend some time figuring out what metrics are appropriate for your organization, which really means figuring out your organization’s purpose or product.

Metrics are going to be very different for various parts of an organization. They have to be, because otherwise organizations would be monolithic. If there is some kind of grouping of purpose in an organization, that grouping is going to provide your first principles for figuring out what metrics to keep. If you are in the Information Technology (IT) department, your metrics will be related to IT matters. If you’re in Human Resources, your metrics are going to be related to HR subjects, and so forth. The place where you would start thinking about monolithic metrics across organizational sub-divisions is if you are an executive trying to do “business intelligence” analysis of how the sub-divisions work with one another.

Generally, your metrics are going to be outcome focused or operations focused, such as how many widgets you produced, or how you produced them. IT security is an interesting problem because security metrics tend to focus on the difference between what is working right now, and what is likely to break. We’re not dealing with a simple “materials in, widgets out” type of analysis, because we immediately fall into the problem IT security faces; we’re concerned with an unpredictable future. IT security metrics are going to tend to be oriented toward figuring out efficiency, effectiveness and long-term outcomes. That’s why it’s hard!

It’s important to realize there is no “one size fits all metric” because there is no one size fits all organization. There is plenty of room to keep common data-points in our community, such as “number of email messages sent” or “number of users with administrative permissions,” but those are merely interesting details. What we’re trying to do is the hard part – exploring the connection between what security does and what the business does, and trying to quantify it.

What are some of the things security does? Here are a few possibilities:

• Protecting customer data

Advertisement. Scroll to continue reading.

• Protecting our company’s intellectual property

• Managing reputational risk

• Preventing service interruptions

• Responding to incidents

• Deflecting denial of service attacks

• Managing vulnerabilities and configurations

• Assessing, analyzing, and recommending technologies

If you look at the list above, and you see a couple of the items as being ones that IT security “owns” in your organization, you can then ask yourself about the common elements between the items you’re responsible for. Then, start to map them mentally into the way your security department goes about doing them. For example, preventing service interruptions, incident response, and dealing with denial of service attacks are likely to overlap functionally – you may want to analyze that capability as a single unitary function. Technology strategy and the highly nebulous goal of “protecting customer data” may be a meta-function of supporting what amounts to a consultative practice for executives and business units.

Don’t worry, we’re not going to start slapping measurements on something as vague-sounding as “protecting customer data” – yet – but since it’s part of what you may be responsible for, we need a framework for figuring out what can be measured and how to make it relevant. How do we do that? To start, we need to do a deeper analysis of what you do when you “protect customer data.” What does the process of protecting customer data look like, the way you do it? What sub-steps in that process require which resources and in what quantity? Is that process repeatable enough that it’s somewhat predictable?

In theory, once someone begins this kind of analysis, they may discover that their organization is completely random, and their security process is therefore more or less nonexistent. That’s the theory, but in practice I suspect that anything other than a start-up in its first couple months of operation will already have some kind of established processes that can be analyzed. If you’re reading this and you’re thinking, “we have no processes at all, everything we do is always chaos!” then you may be one of the rare people who works for an organization that isn’t organized at all.

Next up: Establishing your own metrics: a framework for figuring out what to do.

Written By

Click to comment

Daily Briefing Newsletter

Subscribe to the SecurityWeek Email Briefing to stay informed on the latest threats, trends, and technology, along with insightful columns from industry experts.

Join security experts as they discuss ZTNA’s untapped potential to both reduce cyber risk and empower the business.


Join Microsoft and Finite State for a webinar that will introduce a new strategy for securing the software supply chain.


Expert Insights

Related Content

Application Security

Cycode, a startup that provides solutions for protecting software source code, emerged from stealth mode on Tuesday with $4.6 million in seed funding.


A recently disclosed vBulletin vulnerability, which had a zero-day status for roughly two days last week, was exploited in a hacker attack targeting the...

Risk Management

The supply chain threat is directly linked to attack surface management, but the supply chain must be known and understood before it can be...

Data Breaches

LastPass DevOp engineer's home computer hacked and implanted with keylogging malware as part of a sustained cyberattack that exfiltrated corporate data from the cloud...

Cybersecurity Funding

2022 Cybersecurity Year in Review: Top news headlines and trends that impacted the security ecosystem

Incident Response

Microsoft has rolled out a preview version of Security Copilot, a ChatGPT-powered tool to help organizations automate cybersecurity tasks.

Endpoint Security

Today, on January 10, 2023, Windows 7 Extended Security Updates (ESU) and Windows 8.1 have reached their end of support dates.

Data Breaches

GoTo said an unidentified threat actor stole encrypted backups and an encryption key for a portion of that data during a 2022 breach.