Security Experts:

Connect with us

Hi, what are you looking for?


Incident Response

Collection and Analysis: Two Sides to the Coin

Sometimes, it seems that talk of “big data”, “security analytics”, and “big data security analytics” is everywhere. I can’t recall the last security conference, professional meeting, or industry event I attended where the topic didn’t come up at least once. In some cases, the topic can even dominate the discourse within a particular forum.

Sometimes, it seems that talk of “big data”, “security analytics”, and “big data security analytics” is everywhere. I can’t recall the last security conference, professional meeting, or industry event I attended where the topic didn’t come up at least once. In some cases, the topic can even dominate the discourse within a particular forum.

One byproduct of all this talk is a confusing and somewhat overwhelming vendor environment for the enterprise buyer. This environment is one where all of the words and ideas can begin to blend together. Since I spent over a decade on the enterprise/operational side before moving to the vendor side, I can sympathize with the confusion this can bring to the enterprise audience. Leaders in the enterprise have many responsibilities, and it is difficult for them to keep track of the large number of vendors and what each vendor’s specialty, focus, and strengths are.

Big Data and Security AnalyticsMany enterprises see the need for and share a desire to be doing “big data” and “security analytics”, and thus, it’s not particularly surprising that many vendors are offering “big data” and “security analytics” solutions.

Let’s take a step back and ask a simple question. What does it actually mean to do “big data” and “security analytics”? As we look to answer this question, I think it’s helpful to think a level deeper about this in order to better understand it.

At a high level, “big data” and “security analytics” are about the two very different, somewhat diametrically opposed, but equally important concepts of collection and analysis. Allow me to explain.

Before it is possible to run analytics, one needs the right data upon which to run those analytics. Before “big data” emerged as a buzzword, this was called “collection” or “instrumentation of the network and endpoint”. Further, in order to run analytics, one also needs a high performance platform upon which to issue the precise, targeted, incisive queries required by analytics. Before “security analytics” emerged as a buzzword, this was sometimes called analysis or forensics, among other terms.

Collection and analysis, at enterprise speeds and volumes, are both equally important. If you think about it, you can’t really have one without the other. Or, to put it another way, what good does the greatest collection capability provide without a way to analyze that data in a timely and accurate manner? Similarly, what good does the greatest analytical capability provide without the underlying data to support it? As we see from these pointed questions, collection and analysis are not only important – they are also highly dependent on one another.

In addition to being the fundamental elements of “big data”, collection and analysis form the cornerstone of a strong security program. Collection and analysis provide an organization with the visibility required to practice Continuous Security Monitoring (CSM).

Although a detailed discussion of CSM is beyond the scope of this post, the topic has been discussed at length by NIST, SANS, Gartner, and others. The goal of CSM is to allow an organization to move rapidly from Detection to Analysis and on to Containment and Remediation. In other words, CSM facilitates and enables the incident response process and life cycle. If we think about it, after a detection event and before we can move to containment and remediation, we need to understand what exactly needs to be contained and remediated.

Analysis of the relevant data we have collected enables us to do that. An organization’s ultimate goal, when prevention efforts fail, is to detect and respond to intrusions before they cause damage to the organization. Without the proper collection and analysis capabilities, it’s not possible to achieve this goal.

Of course, proper Continuous Security Monitoring involves many details. Here are just a few thoughts, some of which I’ve discussed in greater detail in previous SecurityWeek pieces, around strategic steps organizations can take in the area of CSM to improve their information security postures:

• Identification of business risks and concerns to be addressed through Continuous Security Monitoring

• Creation of goals and priorities based on business risks and concerns

• Identification of the least number of data sources of highest value that provide the required visibility across the enterprise

• Collection of those relevant data sources

• Exposure of the collected data with sufficient performance to facilitate Detection, Analysis, Containment, and Remediation

• Development of content and logic leveraging the collected data to supply the work queue with high fidelity alerting

• Development of formalized process for investigation and response

While it may be tempting to collect all of the available data within the enterprise, this actually works against the interests of the security organization. It is prudent to ensure that the minimal data that provides sufficient context and coverage is collected, but not more than that. I’ve discussed this in additional depth in my previous SecurityWeek piece entitled “Incident Response: Focus on Big Value, Not Big Data”. Collecting more data than required creates three high level issues:

• Analytical (query) performance degrades rapidly, making timely incident response nearly impossible

• Retention periods shorten, producing historical blind spots that impede response for long present intrusions

• Confusion, uncertainty, and inefficiency are introduced — the first question during incident response is often “Where do I go to get the data I need?” rather than “What question do I need to ask of the data?”

Big data is an interesting topic with the potential to be an incident response enabler by providing unprecedented visibility.

It’s important to remember that big data involves two equally important but somewhat diametrically opposed interests – collection and analysis. Both aspects are important, but they have a tendency to work against each other if not properly aligned strategically. It’s important to remember the ultimate goal of collection and analysis, which is the enablement of timely incident response. It is in this spirit that we aim to gain the most information from the smallest subset of data. All the data in the world does you no good if you cannot leverage it in a timely manner when you need it most. In incident response, less is more.

Written By

Joshua Goldfarb (Twitter: @ananalytical) is currently a Fraud Solutions Architect - EMEA and APCJ at F5. Previously, Josh served as VP, CTO - Emerging Technologies at FireEye and as Chief Security Officer for nPulse Technologies until its acquisition by FireEye. Prior to joining nPulse, Josh worked as an independent consultant, applying his analytical methodology to help enterprises build and enhance their network traffic analysis, security operations, and incident response capabilities to improve their information security postures. He has consulted and advised numerous clients in both the public and private sectors at strategic and tactical levels. Earlier in his career, Josh served as the Chief of Analysis for the United States Computer Emergency Readiness Team (US-CERT) where he built from the ground up and subsequently ran the network, endpoint, and malware analysis/forensics capabilities for US-CERT.

Click to comment

Expert Insights

Related Content

Application Security

Cycode, a startup that provides solutions for protecting software source code, emerged from stealth mode on Tuesday with $4.6 million in seed funding.

Data Breaches

GoTo said an unidentified threat actor stole encrypted backups and an encryption key for a portion of that data during a 2022 breach.

Management & Strategy

Microsoft making a multiyear, multibillion dollar investment in the artificial intelligence startup OpenAI, maker of ChatGPT and other tools.

Incident Response

Cygnvs emerges from stealth mode with an incident response platform and $55 million in Series A funding.

Management & Strategy

Tens of cybersecurity companies have announced cutting staff over the past year, in some cases significant portions of their global workforce.


Twenty-one cybersecurity-related M&A deals were announced in December 2022.


Thoma Bravo will spend $1.3 billion to acquire Canadian software firm Magnet Forensics, expanding a push into the lucrative cybersecurity business.

Incident Response

A new Mississippi Cyber Unit will be the state’s centralized cybersecurity threat information, mitigation and incident reporting and response center.