Security Experts:

Understanding and Improving the Burden on Threat Hunters

Despite increased security budgets, threat hunters say they are under-resourced and overstretched

By focusing efforts on local threat hunting, organizations are missing the opportunities presented by wider threat reconnaissance. Local threat hunting is the detection and response to threats already existent on the corporate infrastructure. Threat reconnaissance is the detection and avoidance of evolving threats across the wider internet before they reach the home network.

This is one of the key suggested takeaways from Team Cymru’s recent report, The State of Threat Hunting and the Role of the Analyst. The cyber threat intelligence firm surveyed 1,778 IT and IT security professionals in North America, Latin America, the UK and Europe.

The report seeks to “track the level of maturity at which analyst teams and threat hunting teams are operating,” It defines maturity as the use of threat intelligence, internal threat hunting, and external threat reconnaissance. This is a simplistic and subjective model. Different companies have proposed other more detailed maturity models, but usually without including external reconnaissance as a separate aspect.

Most threat hunting teams do some form of external threat reconnaissance; but few do it to the extent recommended by Team Cymru. The failure to adequately define and segregate the two aspects of threat hunting within the questions asked by the survey make it impossible to accurately quantify the extent and value of external surveillance employed. Tracking maturity levels to the definition defined by the report is consequently impossible.

That said, the report (PDF) provides extensive insight into the existing role of corporate threat hunting – showing where threat hunters feel they are failing, and providing pointers on how their value can be improved.

Team Cymru’s premise is that cybersecurity defense is best served by a combination of internal threat hunting (to detect attackers who might be probing the network or are already inside the network), and external threat reconnaissance (to understand and get ahead of the threat actors who are planning to attack the network). The latter is like the high pressing football team that defends its own goal by keeping the ball in the attacker’s half, preventing attacks rather than just defending against attacks.

What the report demonstrates, however, is that the overall efficiency of internal threat hunting still leaves much to be desired. The precise percentage responses to different questions can be found in the report itself, but in general, threat hunters believe that they are too under-resourced and overstretched to provide the best possible service.

This is despite a major increase in security budgets over the last few years. According to the survey, the extrapolated value of this year’s IT budget across the range of responders is $117 million. Within this overall budget, the same process suggests that security activities get 19% of the IT budget, and that analyst activities and threat intelligence receive 22% of the security budget.

If the threat hunters are still under-resourced, the question is whether this remains a budgetary issue, or a consequence of the infamous skills gap.

David Monnier is a Team Cymru Fellow, and senior manager of infrastructure and services. “I would argue that threat hunting problems are more a consequence of the skills gap than the budget,” he told SecurityWeek. “The skills gap remains an issue in all facets of technology. We are probably a generation or two away from the time when society will really understand the technology we rely on. For example, threat hunting is not really a physical talent where you can see things happening, it involves abstract thinking.” The ability to ‘see’ problems and solutions in a list of digital occurrences rather than painstakingly search for them is not currently a natural talent.

However, if we can train or acquire a greater number of natural internal threat hunters, we will not then need to find additional external threat reconnaissance personnel. “External reconnaissance requires a different skill set to internal threat hunting, but it takes an understanding of the same skill set that can scale up,” commented Monnier. 

Threat hunting provides the analyst with a very granular view of the current situation; reconnaissance provides a more disparate wider view available from multiple sources. “For example,” continued Monnier, “internal threat hunting might involve examination of a limited number of IP addresses, where external reconnaissance would involve seeing patterns within the wider DNS. I would class the skills as being part of the same skills pool. It’s not so much different skills as a different application of those skills”

This may provide a possible route to a partial solution to the threat hunting resource problem – not so much in finding new natural threat hunters, but in releasing the existing pool from other responsibilities. Today, too many hunters also double as responders. In an ideal world, the hunters would find and triage an incident, and then hand it off to a responder to handle. 

“Statistically,” said Monnier, “in most companies the hunter is the same person as the responder. Ideally it shouldn’t be like that because of the difference in overall mental strategy. Threat hunting is an offensive action while response is a defensive action.” It’s the same difference as that between red teamers and blue teamers ‒ they are different approaches that require different mentalities. 

“Right now,” he continued, “manpower constraints are so significant that most security teams are understaffed, and as a result that kind of ideal specialization is hard to realize. This is different in the largest corporations that have large practitioner teams. where there is segregation between the hunters and the policy enforcers. That’s the ideal approach.”

Blue teamers are probably easier to find and cheaper to train than red teamers. It would consequently place a lesser strain on the security budget to recruit more responders and to release the existing threat hunters to concentrate on their primary function ‒ perhaps even providing the ability to look beyond internal threat hunting to include external reconnaissance.

This would have a further feedback effect on the internal threat hunting. Team Cymru’s premise is that external reconnaissance is likely to reduce the impact on internal threat hunting. “Once you’ve identified threat actors who are focused on or targeting you specifically, but before they launch the attack,” Monnier explained, “you can start to understand their motives and their tools and tactics, and can start to adjust your posture proactively. You can start to be much more effective at defending against the things that are actively looking for you as opposed to the common drive-by-scan-and-exploit internet attacks.”

These scan and exploit attacks fill up massive logs every day that can become a distraction ‒ which skilled attackers often use to hide within and keep practitioners from identifying the real threat. “These drive-by threats are more likely to be recognized and stopped by your basic policies with the assistance of machine learning detection,” continued Monnier. “But if you can have your team tracking the advanced groups who are targeting you and are more likely to breach your standard defense policies, that will make a more significant impact in protecting you from the most likely successful threats.”

The weakness in the report is that the suggested takeaways are not actually justified by the report. It seeks to demonstrate that the addition of intelligent external surveillance to internal threat hunting would be beneficial. This may be and probably is self-evident but is not proven. Nevertheless, the analysis of the current state of internal threat hunting portrays a discipline with huge potential for improvement, and can pinpoint indicators that can be used to improve the effectiveness of existing threat hunting teams.

Related: IBM Gifts Threat Hunting Tool to Open Cybersecurity Alliance

Related: Creating an Effective Threat Hunting Program with Limited Resources

Related: Attackers Hide in Plain Sight as Threat Hunting Lags: Report

Related: Threat Hunting Tips to Improve Security Operations

view counter
Kevin Townsend is a Senior Contributor at SecurityWeek. He has been writing about high tech issues since before the birth of Microsoft. For the last 15 years he has specialized in information security; and has had many thousands of articles published in dozens of different magazines – from The Times and the Financial Times to current and long-gone computer magazines.