Connect with us

Hi, what are you looking for?


Malware & Threats

Anti-Malware Testing – Industry Insight

Anti-Malware Testing – David Harley

Anti-Malware Testing – David Harley

Journalist/blogger Kevin Townsend recently posed some interesting questions concerning the Anti-Malware Testing Standards Organization (AMTSO). Specifically, he asked, “Is AMTSO the anti-malware industry looking after itself?” The answer is yes, at least in part. Bad test results are bad for a security company’s bottom line. But that’s not the only interest AMTSO has in testing. Good testing helps to improve products, while bad testing tends to promote bad products and cause unwarranted damage to good products, and that isn’t really good for consumers. Maybe only shareholders care about the financial health of an anti-malware company, but wouldn’t you rather have reliable information about the product you choose to protect your systems?

Anti-Malware Testing Standards Association

Townsend is not the first person to suspect AMTSO of putting the fox in charge of the hen house. Security Curve subsequently went so far as to describe AMTSO’s membership list as a Who’s Who of AV vendors, though the 2008 list it quoted actually included several non-vendors, including a couple of the most influential mainstream testers. What’s more, the number of non-vendors has increased since then.

In fact, Townsend and Security Curve are concerned with slightly different issues. Security Curve sees it as a problem that there are any vendors in AMTSO. Indeed, if AMTSO were really a standards organization in the same sense as ISO, say, I have to agree that it would be inappropriate to have vendor members approving or criticizing the way in which testers test their own products. However, I’d hope and expect that such an organization would solicit input from testers and vendors, the best of whom have considerable (and complementary) expertise and experience.

Kevin Townsend, on the other hand, sees testers and vendors as two sides of the same coin, and he has a point. Mainstream vendors and mainstream testers do have a symbiotic relationship, and there’s more to it than the fact that testers need products to test, and individual vendors need someone to provide potential customers with data to prove their products are best.

No-one has a monopoly on the huge totality of known malware samples. (Somewhere around 40 million at time of writing. At least, that’s the rough figure most researchers were bandying around at the CARO workshop in Helsinki in May, which I suppose means it’s significantly larger by now.) Research labs have been seeing tens of thousands of new, unique malicious binaries per day for a while, and occasionally the far side of 100,000. Numbers this large generate all sorts of interesting problems, not only in terms of testing (I’ll come back to that one in another article) but simply in terms of a rational, manageable exchange of samples, data and metadata.

Testers and vendors alike are wrestling with those issues, and not in isolation. Indeed, some testers outside that magic circle have suggested that sample exchange is somehow “cheating”. But vendors (and testers) have been exchanging samples for many years, not as part of some convoluted plan to bias testing, but because it’s not considered appropriate to refuse to share samples with the intention of maintaining some form of competitive advantage. Townsend’s complaint, however, is that AMTSO doesn’t encourage members of the public to join, or as he rather emotively calls them, “Poor Bloody Infantry” (PBI). I understand that users of security software may feel as if they’re perpetually dodging bullets, but it wasn’t the security industry that sent them off to the trenches.

But he does have a point. AMTSO is intended, among other things, to inform and to educate (as per its charter) but it doesn’t particularly engage directly with the public. AMTSO isn’t a for-profit business, but it’s expensive to run, so its subscription fees are too high to interest most individuals.

Advertisement. Scroll to continue reading.

I’m not saying that the opinion of “the masses” doesn’t matter. Actually, customers’ wallets have had as much influence on the shape of the security industry as the research community, and maybe more. But (and I don’t think I can think of a way to see this that won’t sound elitist) the representatives of member organizations who are most active in the organization are experts in their fields. Right now, I’d say that most people need to learn more about how testing currently works before they can contribute usefully to the debate on how to improve it.

Personally, I’d like to see AMTSO engage more with Townsend’s PBI, even if getting the whole population of the Internet onto the same page is a little too ambitious a target right now. It couldn’t be a “free for all.” There has to be some way of keeping down the signal-to-noise ratio. And I don’t think giving Joe User a voice is the same as giving him a vote – I can’t think of a surer way for a voluntary organization to bog down both procedurally and in debate. But I don’t think a dialogue, perhaps on the basis of a second-tier, cheap basic membership fee, is too much to ask, and one way of implementing it would be along the lines of the Anti-Phishing Working Group’s basic $50 membership. But don’t ask me whether or when it will happen. That depends on the views of the existing membership. Interestingly, though, some rather mixed publicity over the past few days has resulted in several applications for individual membership, so perhaps that whole issue might be addressed at the next AMTSO workshop in October.

Written By

Click to comment

Daily Briefing Newsletter

Subscribe to the SecurityWeek Email Briefing to stay informed on the latest threats, trends, and technology, along with insightful columns from industry experts.

Join security experts as they discuss ZTNA’s untapped potential to both reduce cyber risk and empower the business.


Join Microsoft and Finite State for a webinar that will introduce a new strategy for securing the software supply chain.


Expert Insights

Related Content


The changing nature of what we still generally call ransomware will continue through 2023, driven by three primary conditions.


A recently disclosed vBulletin vulnerability, which had a zero-day status for roughly two days last week, was exploited in a hacker attack targeting the...


No one combatting cybercrime knows everything, but everyone in the battle has some intelligence to contribute to the larger knowledge base.

Malware & Threats

Threat actors are increasingly abusing Microsoft OneNote documents to deliver malware in both targeted and spray-and-pray campaigns.

Malware & Threats

Unpatched and unprotected VMware ESXi servers worldwide have been targeted in a ransomware attack exploiting a vulnerability patched in 2021.


The recent ransomware attack targeting Rackspace was conducted by a cybercrime group named Play using a new exploitation method, the cloud company revealed this...

Application Security

Virtualization technology giant VMware on Tuesday shipped urgent updates to fix a trio of security problems in multiple software products, including a virtual machine...

Malware & Threats

A vulnerability affecting IBM’s Aspera Faspex file transfer solution, tracked as CVE-2022-47986, has been exploited in attacks.