Security Experts:

Anti-Malware Testing - Industry Insight

Anti-Malware Testing - David Harley

Journalist/blogger Kevin Townsend recently posed some interesting questions concerning the Anti-Malware Testing Standards Organization (AMTSO). Specifically, he asked, “Is AMTSO the anti-malware industry looking after itself?” The answer is yes, at least in part. Bad test results are bad for a security company’s bottom line. But that’s not the only interest AMTSO has in testing. Good testing helps to improve products, while bad testing tends to promote bad products and cause unwarranted damage to good products, and that isn’t really good for consumers. Maybe only shareholders care about the financial health of an anti-malware company, but wouldn’t you rather have reliable information about the product you choose to protect your systems?

Anti-Malware Testing Standards Association

Townsend is not the first person to suspect AMTSO of putting the fox in charge of the hen house. Security Curve subsequently went so far as to describe AMTSO’s membership list as a Who’s Who of AV vendors, though the 2008 list it quoted actually included several non-vendors, including a couple of the most influential mainstream testers. What’s more, the number of non-vendors has increased since then.

In fact, Townsend and Security Curve are concerned with slightly different issues. Security Curve sees it as a problem that there are any vendors in AMTSO. Indeed, if AMTSO were really a standards organization in the same sense as ISO, say, I have to agree that it would be inappropriate to have vendor members approving or criticizing the way in which testers test their own products. However, I’d hope and expect that such an organization would solicit input from testers and vendors, the best of whom have considerable (and complementary) expertise and experience.

Kevin Townsend, on the other hand, sees testers and vendors as two sides of the same coin, and he has a point. Mainstream vendors and mainstream testers do have a symbiotic relationship, and there’s more to it than the fact that testers need products to test, and individual vendors need someone to provide potential customers with data to prove their products are best.

No-one has a monopoly on the huge totality of known malware samples. (Somewhere around 40 million at time of writing. At least, that’s the rough figure most researchers were bandying around at the CARO workshop in Helsinki in May, which I suppose means it’s significantly larger by now.) Research labs have been seeing tens of thousands of new, unique malicious binaries per day for a while, and occasionally the far side of 100,000. Numbers this large generate all sorts of interesting problems, not only in terms of testing (I’ll come back to that one in another article) but simply in terms of a rational, manageable exchange of samples, data and metadata.

Testers and vendors alike are wrestling with those issues, and not in isolation. Indeed, some testers outside that magic circle have suggested that sample exchange is somehow “cheating”. But vendors (and testers) have been exchanging samples for many years, not as part of some convoluted plan to bias testing, but because it’s not considered appropriate to refuse to share samples with the intention of maintaining some form of competitive advantage. Townsend’s complaint, however, is that AMTSO doesn’t encourage members of the public to join, or as he rather emotively calls them, “Poor Bloody Infantry” (PBI). I understand that users of security software may feel as if they’re perpetually dodging bullets, but it wasn’t the security industry that sent them off to the trenches.

But he does have a point. AMTSO is intended, among other things, to inform and to educate (as per its charter) but it doesn’t particularly engage directly with the public. AMTSO isn’t a for-profit business, but it’s expensive to run, so its subscription fees are too high to interest most individuals.

I’m not saying that the opinion of “the masses” doesn’t matter. Actually, customers’ wallets have had as much influence on the shape of the security industry as the research community, and maybe more. But (and I don’t think I can think of a way to see this that won’t sound elitist) the representatives of member organizations who are most active in the organization are experts in their fields. Right now, I’d say that most people need to learn more about how testing currently works before they can contribute usefully to the debate on how to improve it.

Personally, I’d like to see AMTSO engage more with Townsend’s PBI, even if getting the whole population of the Internet onto the same page is a little too ambitious a target right now. It couldn’t be a “free for all.” There has to be some way of keeping down the signal-to-noise ratio. And I don’t think giving Joe User a voice is the same as giving him a vote – I can’t think of a surer way for a voluntary organization to bog down both procedurally and in debate. But I don’t think a dialogue, perhaps on the basis of a second-tier, cheap basic membership fee, is too much to ask, and one way of implementing it would be along the lines of the Anti-Phishing Working Group’s basic $50 membership. But don’t ask me whether or when it will happen. That depends on the views of the existing membership. Interestingly, though, some rather mixed publicity over the past few days has resulted in several applications for individual membership, so perhaps that whole issue might be addressed at the next AMTSO workshop in October.

view counter
David Harley CITP FBCS CISSP is Research Fellow and Director of Malware Intelligence at ESET LLC , Chief Operations Officer at AVIEN, and on the Board of directors of AMTSO. He is a prolific blogger and author of security-focused conference papers and articles. His books include “Viruses Revealed” (Osborne/McGraw-Hill) and the “AVIEN Malware Defense Guide” (Syngress.) He joined ESET's Research team in January 2008 as Research Author, and was appointed Director of Malware Intelligence in August 2008.