Security Experts:

How to Evaluate Threat Intelligence Vendors That Cover the Deep & Dark Web

Deep & dark web (DDW) communities have long been must-have data sources for threat intelligence programs, but only recently has the market caught up with this need. Organizations in search of a threat intelligence vendor that covers the DDW are now met with so many options that evaluating—much less choosing—a vendor can be an overwhelming process fraught with decision fatigue and often further complicated by misleading marketing. So how do you select the right vendor? Asking the following questions during your evaluation can help:

What types of sources does your collection strategy cover?

The myriad DDW communities where illicit activity occurs vary widely in terms of exclusivity, accessibility, the threat actors who frequent them, and, consequently, the vendors that collect data from them. Not all vendors cover the same breadth and quality of DDW sources, so it’s crucial to ensure a vendor’s collection strategy includes, at a minimum, the following: 

• Illicit marketplaces, as well as specialty shops such as card and account shops

• Communities frequented by threat actors who operate in languages other than English

• Closed sources including invitation-only and password-protected communities 

• Demonstrated ability to discover and collect from new sources and technologies as threat actors adopt them

• Sources relevant to your function, organization, industry, and intelligence requirements (IRs). For fraud teams in the financial industry seeking to reduce fraud losses, these sources might entail card and account shops, forums frequented by fraudsters, and communities known to facilitate threats to financial institutions, for example.

Keep in mind that although the deep web and dark web are often conflated, they aren’t the same thing. The deep web is the expansive region of sites traditional search engines cannot access, while the dark web is a subcomponent of the deep web only accessible via specialized browsing software. Both harbor illicit sites, but in general, dark web sites are fewer and, in most cases, more easily accessible than their deep web counterparts, which more likely vet members. 

Because all dark web sites are technically deep web sites but not vice versa, it’s also important to ensure a vendor covers not just dark web sources—but also deep web sources that fall outside the dark web.    

What roles do human analysts and automation play in your collection strategy?

Another indicator of the breadth and quality of sources a vendor covers is the mixture of automation and human-powered analysis fueling its collection strategy. Most vendors automate collection to some degree, but a collection strategy that relies entirely on automation in place of human analysts likely signifies limitations in a vendor’s DDW source coverage.

Indeed, automated collection is generally only feasible for some of the larger dark web marketplaces, lower-tier forums, and other relatively accessible sources that comprise a mere fraction of the DDW communities in which illicit activity occurs and valuable data are present. Collecting data from more-exclusive sources necessitates a caliber of human expertise that automation alone can’t mimic.

What sorts of backgrounds and expertise do your analysts have? 

A vendor needs a full-time team of highly skilled analysts in order to obtain and analyze DDW data effectively. This is especially critical for data from closed sources such as invite-only and password-protected forums, many of which don’t operate in English. Gaining access typically requires analysts fluent in languages such as Russian, Arabic, Mandarin, Turkish, Farsi, Spanish, French, and Portuguese and intimately familiar with the slang, idioms, and social and cultural norms unique to these exclusive communities.

Analysts should also have the proper domain expertise for supporting your intelligence requirements (IRs), whether they pertain to fraud, cybersecurity, physical security, counterterrorism, insider threat, and/or third-party risk, among many others.

To what extent do you engage with threat actors?

Satisfying certain IRs and objectives may warrant direct interaction with threat actors in DDW communities. It’s important that a vendor can support these inherently difficult and risky engagements, which typically require: 

• Analysts with deep experience engaging directly and successfully with threat actors on behalf of customers

• Persona management and non-attributable infrastructure that facilitate these engagements and related virtual operations safely and at scale

• Detailed playbooks that govern these engagements and document intelligence gains and losses

How does your coordinated disclosure process work?

Most vendors that cover the DDW regularly come across everything from stolen data and insider recruitment, to emerging cyber and physical threats. And in many cases, vendors may identify vulnerabilities, imminent threats, and/or incidents affecting an organization before it does.

Immediately disclosing such findings to the public can be tempting, but doing so can exacerbate the risks for victims. Practicing coordinated disclosure can help reduce these risks, which is why vendors must have, and follow, a formal process for doing so. Although its execution can vary based on the nature and severity of a finding, this process should entail immediately notifying victim organizations and being judicious about if, when, and how any related information is disclosed publicly.

To what extent do you share information with the broader community? 

Regularly sharing information externally signifies a vendor is committed to not only enhancing the security of the broader community, but also to staying informed of the latest insights and research discoveries uncovered by trusted third-parties. 

Many vendors are involved with the ISACs, for example, enabling them to rapidly share, and be notified of, new information on emerging threats, vulnerabilities, and mitigation tactics pertaining to specific industries—and often to the benefit of their customers, among others.

How do you engage with law enforcement?

By covering the DDW, a vendor is often in virtual proximity to illicit actors and activities of interest to law enforcement. Aside from having close relationships and alignment with the appropriate agencies and jurisdictions, it’s essential for vendors to have and adhere to established protocol for sharing pertinent information with law enforcement. 

Vendors without these protocols in place are more likely to engage in duplicative efforts that not only can hinder their ability to support customers’ IRs effectively and efficiently, but in certain cases may also impede law enforcement operations.

Which laws and regulations govern your data gathering and processing methodologies, and to what extent are you compliant?

There are a number of legislative and regulatory requirements dictating the circumstances and procedures with which organizations that gather and process personal data—including vendors that do so within the DDW—must comply. GDPR is perhaps the best-known example that applies to many vendors.

But since requirements tend to vary by country, it’s crucial when evaluating a vendor to understand the legislative and regulatory environment in which it operates. One indicator that a vendor is likely well-established and prioritizes compliance is leveraging legal counsel with data privacy qualifications.

Ultimately, the avalanche of threat intelligence vendors now claiming to cover the DDW is a promising sign that more organizations are recognizing the need to monitor these regions of the Internet. But at the same time, it’s crucial to remember that accessing and obtaining DDW data safely and effectively in support of an intelligence operation requires highly specific skills and infrastructure that, unfortunately, not all vendors possess. Conducting a thorough evaluation is imperative, and the questions outlined here can serve as a starting point.

view counter
Josh Lefkowitz is the CEO of Flashpoint, which delivers Business Risk Intelligence (BRI) to empower organizations worldwide with meaningful intelligence and information that combats threats and adversaries. Lefkowitz has worked extensively with authorities to track and analyze terrorist groups. He has also served as a consultant to the FBI's senior management team and worked for a top tier, global investment bank. Lefkowitz holds an MBA from Harvard University and a BA from Williams College.