Researchers have created custom malware samples in an effort to test the effectiveness of some top advanced persistent threat (APT) attack detection appliances.
APT campaigns are increasingly common and since they usually rely on tools that are not detected by regular antivirus products, many security companies have developed specialized solutions designed to identify and block such threats. The list of firms that offer such solutions includes Cisco, Damballa, Checkpoint, FireEye, Fortinet, Palo Alto Networks, LastLine, Zscaler, Trend Micro and Websense.
Over the past years, independent testing firm NSS Labs has conducted several tests comparing the top solutions, but in many cases the results have been controversial. NSS Labs’ testing methods have been criticized by some vendors whose products did not perform as expected, including FireEye and Palo Alto Networks. Last week, Miercom published the results of a test comparing APT detection solutions from Zscaler and FireEye. In that particular test, Zscaler performed better, but FireEye contested the accuracy of the results and testing methodology.
Researchers from MRG Effitas, a UK-based independent IT security research company focusing on efficacy assessment and assurance services, and CrySyS Lab, the Hungary-based organization that has been involved in the analysis of numerous APT campaigns, have tested five “cutting-edge” solutions. The tested products are all “well-established in the market,” but they have not been named.
“[Our] goal was simply to implement some ideas we had for bypassing cutting-edge APT attack detection tools without actually being detected, and to test if our ideas really work in practice,” researchers wrote in their report.
For their tests, researchers used four samples, which they created over a period of two weeks without having access to any APT attack detection solutions. The samples were designed to incorporate functionality that is typical to remote access Trojans (RATs), including remote code execution, and file download/upload capabilities. No lateral movement was initiated in the tests, but real attacks were simulated through command execution and file transfers.
The first test sample, which used a known Java exploit for delivery, was designed to mimic an attacker with limited knowledge and resources. Samples 2 and 3, which used a Java self-signed applet and a Visual Basic macro for delivery, simulated attackers with moderate knowledge and resources. The last sample, which the researchers dubbed “Babo” (the Hungarian word for “hobbit”), was simple but stealthy.
The first two test samples were detected by all five APT attack detection products. The third sample was detected by only one product, while Babo bypassed all solutions. The authors of the report noted that even though the first two samples were picked up by all solutions, some of them rated the threats as “low severity.”
“The main message of this work is that novel anti-APT tools can be bypassed with moderate effort. If we were able to develop samples that were not detected by these tools without actually having access to any of the tested products during the development phase, then resourceful attackers who may be able to buy these products will also be able to develop similar samples, or even better ones,” the report reads.
The researchers advise organizations that want to acquire APT detection solutions to conduct proper testing, either in-house or outsourced, and not judge the products based on the way they are marketed or their prices.
The complete report, titled “An independent test of APT attack detection appliances,” is available in PDF format.
