MITRE Corporation’s ATT&CK framework has been used to evaluate enterprise security products from several vendors to determine how efficient they are in detecting and responding to attacks launched by sophisticated threat groups.
MITRE is a not-for-profit company involved in federally funded research and development projects in various areas, including cybersecurity. Its Adversarial Tactics, Techniques, and Common Knowledge (ATT&CK) Matrix is a framework that describes the techniques used by adversaries, including related to persistence, privilege escalation, defense evasion, credential access, discovery, data collection, lateral movement, command and control, and execution.
Earlier this year, the company announced that it was giving cybersecurity solutions providers the chance to evaluate their products based on the ATT&CK framework, specifically a plan that emulates activities known to have been carried out by a threat group tracked as APT3.
APT3, linked by researchers to the Chinese Ministry of State Security (MSS), is also known as UPS Team, Gothic Panda, Buckeye and TG-0110. The group has been active since at least 2009, targeting organizations in the United States and elsewhere via spear-phishing, zero-day exploits, and various other methods.
In the first round of evaluations performed by Mitre, the threat actor’s tactics and techniques were tested against products from Carbon Black, CrowdStrike, CounterTack, Endgame, Microsoft, RSA and SentinelOne.
Unlike other evaluations, the MITRE ATT&CK testing does not provide scores or ratings and it’s difficult to make a direct comparison between the products using the results. Instead, the goal is to show how a product’s capabilities can detect different types of malicious activities typically carried out by hackers once they have gained access to an organization’s systems. The tests are also designed to help vendors improve their tools and coverage.
For example, the results of the evaluation show how one of the products failed to detect that the attacker used the Cobalt Strike tool to steal access tokens and escalate privileges. A different product managed to detect the attempt using a combination of telemetry and behavior indicators.
It’s also worth noting that MITRE openly communicates with vendors during these evaluations.
“We announce the techniques as they are executed, and the vendor can ask us details about how the procedures were implemented,” MITRE’s Frank Duff explained. “The vendor then shows us their detections and describes their process so that we can verify the detection. Since our goal is to capture different detection methods, we may even suggest to the vendor how their capability might have detected the behavior.”
MITRE pointed out that its evaluations focus on the technical ability of a product to detect malicious behavior.
“There are other factors we are not accounting for in our evaluations that should be considered by decision makers as they decide which tool best fits their needs,” Duff said. “You should consider factors such as cost of ownership, sophistication of your Security Operations Center, environmental noise, integration with other tools, user interface, security policies, and other factors. One product may not fit every need, and products can address different needs in different ways.”
Related: MITRE ATT&CK Matrix Used to Evaluate Endpoint Detection and Response Product