Connect with us

Hi, what are you looking for?

SecurityWeekSecurityWeek

Incident Response

Intelligence is Not a Numbers Game

If you’ve ever dabbled in data analytics, product design, or digital marketing, you’re likely familiar with vanity metrics. Just as their name implies, vanity metrics are numbers that look good on paper and appear to indicate performance or value. In reality, however, these numbers are deceptive, trivial, and far less useful than meets the eye.

If you’ve ever dabbled in data analytics, product design, or digital marketing, you’re likely familiar with vanity metrics. Just as their name implies, vanity metrics are numbers that look good on paper and appear to indicate performance or value. In reality, however, these numbers are deceptive, trivial, and far less useful than meets the eye.

For example, let’s say that a blog sees a 500 percent increase in page views within a 24-hour period. Such a figure sounds impressive. But it turns out that the blog had launched just the day before, so this figure, though still an accurate percentage, reflects an increase from a mere 10 page views to 50. This scenario reinforcses why page views is a vanity metric—they can be easily manipulated to sound impressive but reveal little about how a website is actually performing.

Although the term itself is lesser-known outside the types of business functions I mentioned above, the underlying concept of vanity metrics persists across the enterprise—including among intelligence programs. 

Indeed, many of the key performance indicators (KPIs) intelligence programs often use to evaluate their operations aren’t really KPIs at all because they don’t accurately indicate performance—they’re simply vanity metrics masquerading as KPIs. Common examples include:

– Number of intelligence reports written

– Number of indicators of compromise (IoCs) processed

– Number of data points collected

– Number of keyword alerts received

Advertisement. Scroll to continue reading.

The problem with relying too heavily on vanity metrics like these is that they can distort our perception of how an intelligence operation is performing. For example, let’s say that the objective of our operation is to reduce fraud losses. We might be tempted to measure its progress by tracking, among other metrics, the number of data points collected in support of the operation. But while a large or larger-than-average number of data points collected would look good on paper and might even suggest increased collections efficiency, it would not definitively indicate that the operation was successful in achieving its objective of reducing fraud losses.

In order to accurately evaluate such an operation, we’d need to choose KPIs that 1) map to the operation’s requirements and objectives; 2) we can feasibly measure at scale; and 3) will help us identify any blind spots or areas for improvement that could arise. Rather than the number of data points collected, the right KPIs for this operation might include:

– Number of emerging fraud schemes identified

– Number of new anti-fraud measures informed

– Number of fraud attempts that were prevented versus the number that were successful

– Relative reduction in fraud losses

Aside from distorting our perception of operational performance, however, vanity metrics can also distort our perception of value—particularly with respect to vendor offerings. Many intelligence programs obtain the data, intelligence, and tools on which their operations rely from third-party vendors, but choosing the right vendor can be difficult given the abundance of misleading claims in the market—many of which are fueled by vanity metrics.

As I mentioned in one of my previous articles, collection strategies are perhaps the biggest differentiator among intelligence vendors. But because many intelligence consumers and decision-makers aren’t always aware of exactly which types of data and sources are best suited for their operations, they often—and understandably—assume that more is better. This assumption is largely why many vendors choose to highlight the following vanity metrics in order to further differentiate their offerings:

– Number of data sources 

– Number of total data points

– Number of new data points collected per day

– Number of customers

But similar to how web page views aren’t indicative of website performance, these types of metrics aren’t indicative of the extent to which an intelligence offering is suitable to a program’s needs and objectives. It’s crucial to remember that regardless of how they are marketed or perceived, the quantitative aspects of a vendor’s collection strategy do not reflect the qualitative aspects. This same principle also applies to your operational KPIs. In other words, there’s no point in having billions of data points if those data points aren’t timely, accurate, actionable, and adequately map to your intelligence objectives and requirements.

Written By

Click to comment

Trending

Daily Briefing Newsletter

Subscribe to the SecurityWeek Email Briefing to stay informed on the latest threats, trends, and technology, along with insightful columns from industry experts.

Join the session as we discuss the challenges and best practices for cybersecurity leaders managing cloud identities.

Register

SecurityWeek’s Ransomware Resilience and Recovery Summit helps businesses to plan, prepare, and recover from a ransomware incident.

Register

People on the Move

Professional services company Slalom has appointed Christopher Burger as its first CISO.

Allied Universal announced that Deanna Steele has joined the company as CIO for North America.

Former DoD CISO Jack Wilmer has been named CEO of defensive and offensive cyber solutions provider SIXGEN.

More People On The Move

Expert Insights

Related Content

Application Security

Cycode, a startup that provides solutions for protecting software source code, emerged from stealth mode on Tuesday with $4.6 million in seed funding.

Cybercrime

A recently disclosed vBulletin vulnerability, which had a zero-day status for roughly two days last week, was exploited in a hacker attack targeting the...

Risk Management

The supply chain threat is directly linked to attack surface management, but the supply chain must be known and understood before it can be...

Data Breaches

LastPass DevOp engineer's home computer hacked and implanted with keylogging malware as part of a sustained cyberattack that exfiltrated corporate data from the cloud...

Cybersecurity Funding

2022 Cybersecurity Year in Review: Top news headlines and trends that impacted the security ecosystem

Endpoint Security

Today, on January 10, 2023, Windows 7 Extended Security Updates (ESU) and Windows 8.1 have reached their end of support dates.

Email Security

Many Fortune 500, FTSE 100 and ASX 100 companies have failed to properly implement the DMARC standard, exposing their customers and partners to phishing...