CONFERENCE On Demand: Cyber AI & Automation Summit - Watch Now
Connect with us

Hi, what are you looking for?


Artificial Intelligence

Bot vs Bot in Never-Ending Cycle of Improving Artificial intelligence

Artificial intelligence, usually in the form of machine learning (ML), is infosecurity’s current buzz. Many consider it will be the savior of the internet, able to defeat hackers and malware by learning and responding to their behavior in all-but real time.

Artificial intelligence, usually in the form of machine learning (ML), is infosecurity’s current buzz. Many consider it will be the savior of the internet, able to defeat hackers and malware by learning and responding to their behavior in all-but real time. But others counsel caution: it is a great aid; but not a silver bullet.

The basic problem is that if machine learning can learn to detect malware, machine learning can learn to avoid detection by machine learning. This is a problem that exercises Hyrum Anderson, technical director of data science at Endgame. 

At the BSides Las Vegas in August 2016 he presented his work on a ‘Deep Adversarial Architectures for Detecting (and Generating!) Maliciousness’. He described the concept of using red team vs blue team gaming, where a ‘malicious’ algorithm continually probes a defensive algorithm looking for weakness, and the defensive algorithm learns from the probes how to improve itself.

Bot vs BotThis week, at the Black Hat conference, Anderson takes the concept further in a presentation titled ‘Testing Machine Learning Malware Detection via Adversarial Probing’. The purpose is simple — to use machine learning to test and improve machine learning defenses. In reality, it is an important step in the continuing battle between attackers and defenders.

Omri Moyal, co-founder and VP of research at Minerva, explains. “Given the increased adoption of anti-malware products that use machine learning, most adversaries will soon arm themselves with the capabilities to evade it,” he told SecurityWeek. “The most sophisticated attackers will develop their own offensive models. Some will copy ideas and code from various publicly-available research papers and some will even use simple trial and error, or replicate the offensive efforts of another group. In this cat-and-mouse chase, the defenders should change their model to mark the evolved attack tool as malicious. A process which is the modern version of ‘malware signature’ but more complex.”

Anderson’s theories will help the defender to stay ahead of the attacker by being both cat and mouse. His Black Hat presentation starts with the understanding that “all machine learning models have blind spots or hallucination spots (modeling error).” At the same time, an advanced attacker knows what models are used by the defender; and can use his own ML to probe for those blind spots.

Moyal explained the implications for defenders. “Just like in previous generations of anti-virus software, attackers can constantly evaluate their malware against the machine learning model until a successful variant is created,” he told SecurityWeek. Malware authors have long tested new or repackaged malware against VirusTotal-like services to see whether it is likely to get past the defenders’ AV defenses. Now they will use ML to test their malware against the known ML defenses, seeking out the blind spots. 

“The resulting specimen,” continued Moyal, “can be used against each victim whose protection relies on this model, offering the attacker a high degree of certainty the malicious program will not be detected. Attackers can also automate this process of generating malware that bypasses the model and even use offensive machine learning to improve this process.”

Anderson’s research is based on the idea of finding the blind spots and closing them before the attackers find them. Ironically, this can be achieved by doing exactly what the attackers will do — use machine learning to probe machine learning. This is nothing more than what security researchers have been doing for decades: probe software to find the weaknesses and get those weaknesses patched before they are found and exploited by the bad people.

Advertisement. Scroll to continue reading.

In today’s presentation, Anderson describes a scientific approach on how to evade malware detection with an AI agent to compete against the malware detector. Although in this instance focusing on Windows PE, the framework is generic and can be used in other domains.

The agent examines a PE file and probes it to find a way to evade the malware detection model. The agent learns how to ‘beat’ the defense. However, as used by the defenders, this approach simply finds the blind spots that can then be fixed. Used solely by attackers, it finds the blind spots that can be exploited.

Anderson’s key takeaway is that machine learning anti malware just bought and installed will offer early success in malware protection, but it will quickly become porous against advanced adversaries. Staying one jump ahead of the bad guys has always been, and remains, the key to infosecurity even in the age of artificial intelligence.

Hyrum Anderson, Bobby Filar, and Phil Roth from Endgame, together with Anant Kharkar from the University of Virginia, have published an associated white paper: Evading Machine Learning Malware Detection (PDF).

Written By

Kevin Townsend is a Senior Contributor at SecurityWeek. He has been writing about high tech issues since before the birth of Microsoft. For the last 15 years he has specialized in information security; and has had many thousands of articles published in dozens of different magazines – from The Times and the Financial Times to current and long-gone computer magazines.

Click to comment

Daily Briefing Newsletter

Subscribe to the SecurityWeek Email Briefing to stay informed on the latest threats, trends, and technology, along with insightful columns from industry experts.

Join us as we delve into the transformative potential of AI, predictive ChatGPT-like tools and automation to detect and defend against cyberattacks.


As cybersecurity breaches and incidents escalate, the cyber insurance ecosystem is undergoing rapid and transformational change.


Expert Insights

Related Content


A recently disclosed vBulletin vulnerability, which had a zero-day status for roughly two days last week, was exploited in a hacker attack targeting the...

Artificial Intelligence

The CRYSTALS-Kyber public-key encryption and key encapsulation mechanism recommended by NIST for post-quantum cryptography has been broken using AI combined with side channel attacks.

Artificial Intelligence

The release of OpenAI’s ChatGPT in late 2022 has demonstrated the potential of AI for both good and bad.

Data Breaches

LastPass DevOp engineer's home computer hacked and implanted with keylogging malware as part of a sustained cyberattack that exfiltrated corporate data from the cloud...

Artificial Intelligence

ChatGPT is increasingly integrated into cybersecurity products and services as the industry is testing its capabilities and limitations.

Artificial Intelligence

The degree of danger that may be introduced when adversaries start to use AI as an effective weapon of attack rather than a tool...

Incident Response

Microsoft has rolled out a preview version of Security Copilot, a ChatGPT-powered tool to help organizations automate cybersecurity tasks.

Data Breaches

GoTo said an unidentified threat actor stole encrypted backups and an encryption key for a portion of that data during a 2022 breach.