Security Experts:

Connect with us

Hi, what are you looking for?


Application Security

Research: Simulated Phishing Tests Make Organizations Less Secure

A large-scale, long-term phishing experiment conducted in a 56,000-employee organization has come to a startling conclusion: Those simulated phishing tests commonly seen in corporate user-education campaigns are actually making things much worse.

A large-scale, long-term phishing experiment conducted in a 56,000-employee organization has come to a startling conclusion: Those simulated phishing tests commonly seen in corporate user-education campaigns are actually making things much worse.

After a 15-month phishing experiment done in partnership with an unnamed publicly traded global company, researchers at ETH Zurich found that embedded training during simulated phishing exercises did not make employees more resilient to e-mail malware lures and, worse, “can have unexpected side effects that can make employees even more susceptible to phishing.”

The results of the landmark experiment was published this week by the Department of Computer Science at ETH Zurich, a public research university based in Switzerland. It calls into question the expanding corporate spend on user education campaigns that combine simulated phishing attacks with training videos and mandatory quizzes.

Even as investors pour money into startups promising respite from phishing attacks, corporate defenders struggle to block sophisticated email lures that serve as the initial entry point debilitating malware and ransomware extortion attacks. For decades, businesses added user awareness training to cybersecurity budgets in attempts to help employees to spot suspicious links or malicious email attachments.

[ PREVIOUSLY:  Investors Race to Find Profits in Securing Corporate Email ]

There has been active debate over the years about the effectiveness of security awareness training but that has not stopped CISOs (Chief Information Security Officers) from flagging user education as a major priority for cyber defense spending.

The ETH Zurich university study is sure to rekindle that debate with the researchers calling for caution in the deployment of methods like embedded phishing exercises and training, warning of “potential negative side effects.” 

The researchers ran the experiment for 15 months (July 2019 through October 2020) where simulated phishing emails were sent to a quarter of the partner company’s workforce during their normal work flow and context. 

The researchers carefully measured click rates from the phishing lures, submission of credentials, and the enabling macros on attachments.  The team also deployed a reporting button to the corporate email client to allow the employees to click and report suspicious emails.

The raw data from the study tell an interesting story:

  • Overall, the study participants clicked on 6,680 out of 117,864 simulated phishes (5.67%). During the 15 months, 4,729/14,733 participants (32.10%) clicked on at least one phish. 
  • The trend for dangerous actions is similar, with the numbers slightly lower: participants fell for 4,885 simulated phishing emails (4.14% of the total sent emails, and 73.13% of all the clicked simulated phishes), and 3,747/14,733 participants (25.43%) users did at least one dangerous action. 
  • There were 4,260 study participants that reported at least one email. In total, the participants reported 14,401 emails, of which 11,035 were our simulated emails. The button to report phishing was also deployed to 6300 employees that were not part of the experiment but could report phishing: 1,543 of them reported at least one suspicious email, and they reported 4,075 emails. Thus, the total number of reported emails we received during the 15 months was 18,476.

[ READ: Security Awareness Training Debate: Does it Make a Difference? ]

The researchers also found “repeated clickers” who fell victim to multiple lure mails and concluded that many employees in an organization “will eventually fall for phishing if continuously exposed.”

In the experiment, the researchers found that 4,729 out of 14,733 (32.10%) participants clicked on at least one link or attachment in simulated phishing emails. A similar high number applies to dangerous actions: 3,747 out of 14,733 (25.43%) performed at least one. 

“These results indicate that a rather large fraction of the entire employee base will be vulnerable to phishing when exposed to phishing emails for a sufficiently long time. We are the first to show such results at scale,” the researchers added.

[ SEE: Report: Security Awareness Training Top Priority for CISOs ]

The study, which was supported by the Zurich Information Security and Privacy Center (ZISC), also found that crowd-sourced phishing detection can be effective and practical in large organizations. 

“Our experiment shows that crowdsourced phishing detection enables organizations to detect a large number of previously unseen real phishing campaigns with a short delay from the start of the campaign,” the team said, noting that the operational load of phishing report processing can be kept small, even in large organizations. 

“Our study also demonstrates that a sufficiently high number of employees report suspicious emails actively over long periods of time. In summary, we show that crowd-sourced phishing detection provides a viable option for many organizations.”

Related: Investors Race to Find Profits in Securing Corporate Email

Related: Security Awareness Training Debate: Does it Make a Difference?

Related: Report: Security Awareness Training Top Priority for CISOs

Written By

Ryan Naraine is Editor-at-Large at SecurityWeek and host of the popular Security Conversations podcast series. He is a security community engagement expert who has built programs at major global brands, including Intel Corp., Bishop Fox and GReAT. Ryan is a founding-director of the Security Tinkerers non-profit, an advisor to early-stage entrepreneurs, and a regular speaker at security conferences around the world.

Click to comment

Expert Insights

Related Content

Application Security

Cycode, a startup that provides solutions for protecting software source code, emerged from stealth mode on Tuesday with $4.6 million in seed funding.

Data Breaches

GoTo said an unidentified threat actor stole encrypted backups and an encryption key for a portion of that data during a 2022 breach.


Websites of German airports, administration bodies and banks were hit by DDoS attacks attributed to Russian hacker group Killnet

Network Security

NSA publishes guidance to help system administrators identify and mitigate cyber risks associated with transitioning to IPv6.

CISO Strategy

Cybersecurity-related risk is a top concern, so boards need to know they have the proper oversight in place. Even as first-timers, successful CISOs make...


The North Korean APT tracked as TA444 is either moonlighting from its previous primary purpose, expanding its attack repertoire, or is being impersonated by...

Identity & Access

Hackers rarely hack in anymore. They log in using stolen, weak, default, or otherwise compromised credentials. That’s why it’s so critical to break the...

Malware & Threats

Microsoft plans to improve the protection of Office users by blocking XLL add-ins from the internet.