Security Experts:

Collection Strategies: The Key Differentiator Among Threat Intelligence Vendors

The outcome of an intelligence operation depends largely on the data that fuels it. Even the most sophisticated operation will fail to produce intelligence of value if its data is not also of value. This concept highlights the biggest differentiator and most important factor to consider when choosing a threat intelligence vendor: data source coverage and, more specifically, collection strategy.

The following questions are designed to help security practitioners more effectively evaluate threat intelligence vendors’ collection strategies:

What types of sources comprise your collection strategy?

Most vendors’ collection strategies include Deep & Dark Web (DDW) and open web sources, but the manner in which these sources are often described to prospective customers can be confusing at best and misleading at worst. While some vendors are understandably vague about certain aspects of their collection strategy in order to protect their access to more sensitive sources, others are simply too vague, revealing little more than the fact that they collect data from DDW and open-web sources.

Knowing that a vendor collects data from the DDW and/or the open web doesn’t tell you nearly enough about the value and origin of such data. While some of the more popular DDW marketplaces such as Dream Market are accessible to anyone with a Tor browser, for example, private DDW forums are highly exclusive, typically invite-only, and contain data that tends to differ substantially from that which is generally available from other types of sources in the DDW. 

Sources can and should be described and categorized far more granularly than just DDW or open web. Within each of these broad categories exist numerous types of sources containing highly differentiated data that can make all the difference between a failed intelligence operation and a successful one. These sources generally include:

- Private or invite-only forums

- Chat services platforms

- Illicit marketplaces

- Payment card shops

- Paste sites

- Social media sites

Given that both DDW and open web sources tend to be poorly delineated in the market, it’s important to understand specifically what sources comprise a vendor’s collection strategy before you decide to become a customer. 

How does your collection strategy map to my intelligence requirements? 

Regardless of how comprehensive a vendor’s collection strategy is, if it doesn’t sufficiently map to your intelligence requirements (IRs), it’s probably time to consider other vendors. Because IRs lay the foundation and set the direction for the entirety of your intelligence operation, naturally they also dictate the types of data and sources your operation will need in order to be effective. This is why it’s crucial to establish your IRs before evaluating vendors. Once you do, discuss them thoroughly with the vendor. Ask if the vendor has access to sources that map to your IRs, and if the answer is yes, dig deeper with follow-up questions such as:

- Which of your sources would be most suitable for my IRs and why?
- Should you lose access to those sources, are suitable backups available?
- What are some examples of how your collection strategy has supported customers with similar IRs?
- What are your collection strategy’s most substantial weakness or blind spots with respect to my IRs?

Keep in mind that no vendor will have 100% coverage of each and every source that could satisfy your IRs and support your operation, but some vendors will have access to more and better sources than others. 

What role does automation play in your collection strategy?

Most vendors automate collection to some degree. But when automation plays too little or too large a role in a vendor’s collection strategy, it could signify a red flag. In general, sources that are easier to access are easier to collect data from automatically. Open web sources such as paste sites are a case in point; because these sites are openly, freely, and safely accessible to anyone with internet access, most vendors can and do collect data from them automatically.

But if a vendor claims to automate the entirety of its collections, it likely lacks the ability to access and/or accurately analyze data from certain types of sources. Private or invite-only forums, for example, are highly exclusive, extremely difficult to access, and therefore nearly impossible to collect data from automatically. Because many of the adversaries who frequent these forums don’t operate in English, gaining access can only be done by human analysts with the necessary linguistic skills. And in many cases, simply being fluent in Russian, Arabic, Mandarin, Turkish, Farsi, Spanish, French, or other languages isn’t enough—analysts also need a keen understanding of the cultural nuances, social norms, idioms, and slang that exist within such communities. Despite promising advances in artificial intelligence and automation, such tools aren’t yet capable of mimicking the level of human expertise required to collect data from these types of sources. 

It should come as surprise that evaluating a threat intelligence vendor’s collection strategy effectively is a complex process that requires far more than simply obtaining the answers to the questions outlined above. But regardless of your requirements, objectives, or program capabilities, remember that the outcome of your intelligence operation depends largely on the data that fuels it.

view counter
Josh Lefkowitz is the CEO of Flashpoint, which delivers Business Risk Intelligence (BRI) to empower organizations worldwide with meaningful intelligence and information that combats threats and adversaries. Lefkowitz has worked extensively with authorities to track and analyze terrorist groups. He has also served as a consultant to the FBI's senior management team and worked for a top tier, global investment bank. Lefkowitz holds an MBA from Harvard University and a BA from Williams College.