Researchers have created custom malware samples in an effort to test the effectiveness of some top advanced persistent threat (APT) attack detection appliances.
APT campaigns are increasingly common and since they usually rely on tools that are not detected by regular antivirus products, many security companies have developed specialized solutions designed to identify and block such threats. The list of firms that offer such solutions includes Cisco, Damballa, Checkpoint, FireEye, Fortinet, Palo Alto Networks, LastLine, Zscaler, Trend Micro and Websense.
Over the past years, independent testing firm NSS Labs has conducted several tests comparing the top solutions, but in many cases the results have been controversial. NSS Labs’ testing methods have been criticized by some vendors whose products did not perform as expected, including FireEye and Palo Alto Networks. Last week, Miercom published the results of a test comparing APT detection solutions from Zscaler and FireEye. In that particular test, Zscaler performed better, but FireEye contested the accuracy of the results and testing methodology.
Researchers from MRG Effitas, a UK-based independent IT security research company focusing on efficacy assessment and assurance services, and CrySyS Lab, the Hungary-based organization that has been involved in the analysis of numerous APT campaigns, have tested five “cutting-edge” solutions. The tested products are all “well-established in the market,” but they have not been named.
“[Our] goal was simply to implement some ideas we had for bypassing cutting-edge APT attack detection tools without actually being detected, and to test if our ideas really work in practice,” researchers wrote in their report.
For their tests, researchers used four samples, which they created over a period of two weeks without having access to any APT attack detection solutions. The samples were designed to incorporate functionality that is typical to remote access Trojans (RATs), including remote code execution, and file download/upload capabilities. No lateral movement was initiated in the tests, but real attacks were simulated through command execution and file transfers.
The first test sample, which used a known Java exploit for delivery, was designed to mimic an attacker with limited knowledge and resources. Samples 2 and 3, which used a Java self-signed applet and a Visual Basic macro for delivery, simulated attackers with moderate knowledge and resources. The last sample, which the researchers dubbed “Babo” (the Hungarian word for “hobbit”), was simple but stealthy.
The first two test samples were detected by all five APT attack detection products. The third sample was detected by only one product, while Babo bypassed all solutions. The authors of the report noted that even though the first two samples were picked up by all solutions, some of them rated the threats as “low severity.”
“The main message of this work is that novel anti-APT tools can be bypassed with moderate effort. If we were able to develop samples that were not detected by these tools without actually having access to any of the tested products during the development phase, then resourceful attackers who may be able to buy these products will also be able to develop similar samples, or even better ones,” the report reads.
The researchers advise organizations that want to acquire APT detection solutions to conduct proper testing, either in-house or outsourced, and not judge the products based on the way they are marketed or their prices.
The complete report, titled “An independent test of APT attack detection appliances,” is available in PDF format.

Eduard Kovacs (@EduardKovacs) is a contributing editor at SecurityWeek. He worked as a high school IT teacher for two years before starting a career in journalism as Softpedia’s security news reporter. Eduard holds a bachelor’s degree in industrial informatics and a master’s degree in computer techniques applied in electrical engineering.
More from Eduard Kovacs
- AntChain, Intel Create New Privacy-Preserving Computing Platform for AI Training
- Several Major Organizations Confirm Being Impacted by MOVEit Attack
- Verizon 2023 DBIR: Human Error Involved in Many Breaches, Ransomware Cost Surges
- Google Patches Third Chrome Zero-Day of 2023
- Ransomware Group Used MOVEit Exploit to Steal Data From Dozens of Organizations
- Cybersecurity M&A Roundup: 36 Deals Announced in May 2023
- In Other News: Government Use of Spyware, New Industrial Security Tools, Japan Router Hack
- Apple Denies Helping US Government Hack Russian iPhones
Latest News
- KeePass Update Patches Vulnerability Exposing Master Password
- AntChain, Intel Create New Privacy-Preserving Computing Platform for AI Training
- Keep Aware Raises $2.4M to Eliminate Browser Blind Spots
- Google Workspace Gets Passkey Authentication
- Cybersecurity Startup Elba Raises €2.5 Million for Employee-Focused Product
- Zoom Expands Privacy Options for European Customers
- Several Major Organizations Confirm Being Impacted by MOVEit Attack
- Apple Unveils Upcoming Privacy and Security Features
