CONFERENCE On Demand: Cyber AI & Automation Summit - Watch Now
Connect with us

Hi, what are you looking for?


Artificial Intelligence

The $64k Question: How Does AI Phishing Stack Up Against Human Social Engineers?

The Rise of AI in Phishing: Will future phishing attacks that leverage artificial intelligence be more dangerous?

Future AI-generated phishing emails are likely to be more effective and damaging than the email-based attacks we are seeing today.

Since the arrival of ChatGPT, the media and security pundits have warned that phishing is now on steroids: more compelling and a vastly increased tempo. IBM’s X-Force Red wanted an objective assessment on this subjective assumption.

The method chosen was to test an AI-generated phishing email and a human generated email against employees working for a healthcare firm. Sixteen hundred staff members were selected: 800 received the AI phish, while the other 800 received the human phish.

The outcome of the investigation is that AI can produce a phish considerably faster than humans (five minutes from five simple prompts compared to 16 hours for the IBM human social engineers); but that human social engineering is currently more effective than AI phishing.

Stephanie Carruthers, IBM’s Chief People Hacker at X-Force Red, puts human success down to three major factors: emotional intelligence, personalization, and a more succinct and effective headline. “Humans,” notes IBM’s report on the test, “understand emotions in ways that AI can only dream of. We can weave narratives that tug at the heartstrings and sound more realistic, making recipients more likely to click on a malicious link.”

In short, the current algorithmic recompilation of stored knowledge is not as compelling as an OSINT-driven human narrative. 

But this is only half the story.

Firstly, the results were close. The human phish achieved a 14% click rate against 11% from the AI phish. Fifty-two percent of the human emails were reported as suspicious, against 59% of the AI emails.

Secondly, AI is in its infancy while human social engineering has been honed over decades of experience. Two questions: could the AI have been used more efficiently (for example with different prompts; that is, better prompt engineering), and how much will AI improve over the next few years?

Advertisement. Scroll to continue reading.

Carruthers is aware of these issues. “I spent hours creating the prompt engineering and figuring out which ones worked – and I can tell you the first ones I produced were garbage. A lot with AI is garbage in garbage out,” she told SecurityWeek. She is confident that these were the best prompts that could be achieved today. “I think I have very solid principles and techniques with what I was asking it to do… I am very happy with the results.”

One example explains her efforts. ChatGPT can be prompted to answer in different styles. Given the apparent lack of ‘emotional intelligence’, could the AI be instructed to respond with greater emotion? “The first responses I got were good, but felt just a bit robotic, a bit cold,” said Carruthers. She tried to inject warmth. “But the more I played with it the more like it just started to break – it just doubled down on the coldness, or it got really wacky. It was hard to find that balance.”

The second question is the big unknown – how much will AI improve over the next few years? This itself has two parts: how much will publicly available AI improve, and how much will criminal AI improve?

Gen-AI obtains its information from what it ingests. Public gen-AI must be wary in this. It must avoid absorbing dangerous personal information that can then resurface in its responses. Criminal AI has no such concerns. So, while the primary source for public AI will be the surface web (with compliance guardrails), there are no such restrictions for criminal AI – which will most likely combine both the surface and the dark web as its data source, with no guardrails.

The potential for criminal AI to include and combine stolen personal data could lead to highly personalized spear-phishing. If this is combined with improved emotional intelligence, the result is likely to be very different to today’s IBM test.

This is subjective conjecture and is exactly what IBM was trying to avoid in its study. But given that ChatGPT can already achieve an 11% success rate in its phishing, it is not something we should completely ignore.

Carruthers own primary takeaway from her study admits such. “If you had asked me before I started who I think would win, I would say humans, hands down. But the more I started prompt engineering, I started getting a little nervous and… these emails are getting better and better,” she told SecurityWeek

“So, I think my biggest takeaway is to question what the future is going to look like.  If we continue to improve gen-AI and make it sound more human, these phishing emails are going to be possibly devastating.”

Related: Malicious Prompt Engineering With ChatGPT

Related: ‘Grim’ Criminal Abuse of ChatGPT is Coming, Europol Warns

Related: ChatGPT, the AI Revolution, and the Security, Privacy and Ethical Implications

Related: Don’t Expect Quick Fixes in ‘Red-Teaming’ of AI Models. Security Was an Afterthought

Written By

Kevin Townsend is a Senior Contributor at SecurityWeek. He has been writing about high tech issues since before the birth of Microsoft. For the last 15 years he has specialized in information security; and has had many thousands of articles published in dozens of different magazines – from The Times and the Financial Times to current and long-gone computer magazines.

Daily Briefing Newsletter

Subscribe to the SecurityWeek Email Briefing to stay informed on the latest threats, trends, and technology, along with insightful columns from industry experts.

Join us as we delve into the transformative potential of AI, predictive ChatGPT-like tools and automation to detect and defend against cyberattacks.


As cybersecurity breaches and incidents escalate, the cyber insurance ecosystem is undergoing rapid and transformational change.


Expert Insights

Related Content

Artificial Intelligence

The CRYSTALS-Kyber public-key encryption and key encapsulation mechanism recommended by NIST for post-quantum cryptography has been broken using AI combined with side channel attacks.

Artificial Intelligence

The release of OpenAI’s ChatGPT in late 2022 has demonstrated the potential of AI for both good and bad.

Artificial Intelligence

ChatGPT is increasingly integrated into cybersecurity products and services as the industry is testing its capabilities and limitations.

Artificial Intelligence

The degree of danger that may be introduced when adversaries start to use AI as an effective weapon of attack rather than a tool...

Artificial Intelligence

Two of humanity’s greatest drivers, greed and curiosity, will push AI development forward. Our only hope is that we can control it.


The easiest way for a cyber-attacker to gain access to sensitive data is by compromising an end user’s identity and credentials. Things get even...

Application Security

Fortinet on Monday issued an emergency patch to cover a severe vulnerability in its FortiOS SSL-VPN product, warning that hackers have already exploited the...

Fraud & Identity Theft

Famed hacker Kevin Mitnick has died after a battle with pancreatic cancer.  At the time of his death, he was Chief Hacking Officer at...