In May 2016 VirusTotal (VT) changed its rules. Any vendor wishing to receive antivirus results via the VT API would in future be required to integrate its own detection scanner into the public VT interface. Furthermore, such vendors would need to be certified by The Anti-Malware Testing Standards Organization (AMTSO).

At the time it looked like a coup engineered by the first-generation anti-malware industry, AMTSO and VT itself to exclude next-generation (next-gen) endpoint security products from gaining benefit from VT. Since that time, however, at least four next-gen companies have joined AMTSO and agreed to abide by the VT rules. More are likely to follow – and it’s beginning to seem that what appeared to be a declaration of war was actually an invitation to peace.

Background

VirusTotal is commonly seen as a service that simply allows users to test a suspicious file against more than 50 anti-malware scanning engines. This is a valuable service – but the real value of VT is in sharing malicious files with the wider endpoint security industry. This allows any vendor receiving those files to ensure that its own product can detect the malware concerned.

VirusTotal is categorically not a viable method of comparing different endpoint security products. The scanning engines used by the VT public interface do not represent the endpoint product in its entirety. If VT says a particular product does not recognise a malware sample, it does not mean that the fully installed product will not detect it.

At the beginning of this year, VT comprised first-gen vendors alone. However, a new generation of endpoint security products is emerging, largely but not entirely categorized by the use of machine learning technologies to help detect malware. These vendors are young and often aggressive in their search for market share. Some have abused the VT service by subscribing to the malware sample service without supplying VT with their own samples; and by using the VT public interface to unfairly and possibly inaccurately claim that first-gen vendors do not detect known malware that they themselves do detect.

The first-gen vendors have felt increasingly aggrieved, and it is that grievance that made the new VT rules appear to be a coup aimed against the next-gen vendors. That’s not quite what happened.

The Coup that Never Was

VirusTotal, owned by Google, became increasingly concerned that the first-gen vendors would withdraw. Without those vendors, there would be no VirusTotal.

VT’s solution was effectively to pass the buck to AMTSO – and the first AMTSO knew was a VT blog saying that it would only allow vendors that an AMTSO member-tester had certified.

“That took us by surprise,” said AMTSO’s General Manager, Dennis Batchelder. “But it gave us a good opportunity to try to solve this problem: how do you fairly join a multi scanner system and get all the benefits of being able to get the crowdsource malware samples coming in; and how do you do that in a fair way that doesn’t chase away the first-gen or the second-gen, and doesn’t create a fight?”

It was, he added, a classic multi-stakeholder problem ideally suited to an AMTSO solution. AMTSO developed a set of proposals: Recommendations when including detection technologies in multi-scanner services. They were designed to be inclusive rather than exclusive – a roadmap for all endpoint security companies to play fair and benefit from the services of VirusTotal for the good of all customers.

There are four key points: vendors cannot use VT “to market or highlight detection efficacy or deficiencies”; all vendors participate “in balanced two-way sample sharing with others in the industry, preferably with an industry-accessible repository”; all vendors play fair in tests; and that these rules apply equally to all vendors regardless of being first-gen or second-gen vendors.

It is these AMTSO recommendations that form the basis of VT’s new rules. And they seem to be working. At the time of writing, in little more than three months, four next-gen vendors have taken steps to conform: Crowdstrike, Invincea, Carbon Black and Palo Alto Networks. More are expected. Some will hold out, but as more vendors sign up to AMTSO and VT, the ‘outsiders’ will find it increasingly difficult to justify their stance.

Rather than being a declaration of war between the generations, VT’s new rules might well prove to be an olive branch.

First-gen Versus Next-gen: The Technologies

A big problem for the endpoint security market is that there is no uniform definition of what each term means. In practice it comes down to little more than ‘the old anti-malware vendors’ and ‘the new endpoint security vendors’. This is a misleading characterization, since both sets use very similar technology, and both sets seek the same end – the detection, prevention and elimination of malware.

Even then there are some companies that still fall outside of this generalization. Bromium, for example, is a genuinely different next-generation technology. Simon Crosby, CTO and co-founder of Bromium, says, “Bromium uses micro-virtualization, which relies only on endpoint CPU virtualization, to hardware i
solate each task on the endpoint so that if malware executes, it cannot persist, steal data or credentials, or access high value networks or sites.” He has little confidence in any of the ‘detect to protect’ technologies, claiming “these services simply cannot keep up with malware that changes by the minute.”

Cylance is another next-gen company that does not believe it can be compared to first-gen vendors. “Cylance has introduced a paradigm shift in the security industry,” said Chad Skipper, VP of Product Testing and Certification, “utilizing machine learning to prevent advanced and commodity malware from executing on the endpoint.”

This statement is true and false. Machine learning can be significantly different to signature-based detection; but there is probably no single first-gen company that relies solely on signature detection; and the suggestion that they do is misleading. The majority of first-gen companies have included machine learning for the last ten years. Andy Patel, security advisor at first-gen vendor F-Secure, comments, “Our first machine learning system was taken into production use back in 2006, and it took almost 10 years for ‘second’ generation vendors to figure out what we were doing.”

The obvious question is that if both generations of endpoint security vendors are using machine learning to teach their engines how to detect malware, why have first-gen vendors made little use of the term. Patel suggests, “We were silent about machine learning and other AI techniques, because they were so useful that we did not want out competitors to know about them and start their own research on the topic. And any other security company who figured out the usefulness of AI did pretty much the same.”

But while first-gen companies have grown into machine learning, next-gen companies have started with machine learning – and as newcomers they have needed something to differentiate themselves from the existing vendors. Their battlecry that new machine learning second-gen vendors are automatically superior to old signature-based engines is more marketing than reality.

It would be more realistic to stop talking about first-gen and next-gen, and simply call all of them ‘endpoint-security’ products.

Can First-Gen and Next-Gen Vendors Come Together and Live Harmoniously as Endpoint Security Providers?

Both AMTSO and VirusTotal are indicators that competing vendors can collaborate for the overall benefit of their customers and security at large. AMTSO’s leadership team currently comprises members from ESET, Avira, Symantec, AVG and Panda (it would benefit from the rapid inclusion of at least one ‘next-gen’ vendor).

These are all first-gen vendors – an industry that has been central to the security industry for many decades. You could say that it has matured. Andy Patel, again: “First gen security vendors have never been all that worried about new competition arriving on the scene. New companies have popped up over the last three decades, and many of them have carved off a piece of the pie for themselves. New security vendors always bring a fresh take on the subject, with new ideas and new technologies. Cooperation between industry players in things like VirusTotal and independent testing benefits everyone who uses these technologies, and makes the Internet safer.”

Luis Corrons, technical director at first-gen PandaLabs, takes a similar view, suggesting all anti-malware companies will co-exist. “At the end of the day what will happen is that all of them will evolve into something similar, where we won’t be able to distinguish them. A good number of the so called ‘first-gens’ have been using the same kind of technologies as the ‘next gens’ for years.”

These are fairly typical views from the first-gen vendors. Not all next-gens agree. “This all depends upon the first-gen vendors. Cylance can stand on its own accord and our customers continue to replace first generation anti-virus with Cylance.” Cylance told SecurityWeek that it is not opposed to joining AMTSO in the future, and is currently exploring that possibility.

So far, just four next-gen companies have joined AMTSO since VirusTotal’s ultimatum: Crowdstrike, Invincea, Carbon Black and Palo Alto. All take a user and community-centric view to threat detection. “We want to work with the community to contribute to community standards,” said CrowdStrike’s chief scientist Dr. Sven Krasser.

Carbon Black and Palo Alto did not specifically respond to a direct approach for this article; but Invincea did.

Our hope, said Invincea CEO and founder Anup Ghosh, is “that the industry of the next-gen players is maturing. If all you are doing is marketing how great you are but not doing either third-party testing or listing yourself on Virus Total then you are doing a disservice to the user. It’s really saying you’re not willing to stand by your product.”

He goes further, to explicitly agree with the concerns that have been voiced by first-gen vendors. “You have these next-gen companies leeching off the IP of VirusTotal to make their products better but at the same time throwing [first-gen vendors] under the bus. I agree it’s patently unfair; it is a community and Invincea benefits from that community.” Ghosh recognizes the value of being able to train his machine learning against the huge VirusTotal resource of malware samples, and the fairness of contributing back to that shared resource.

First-gen’s two primary grievances

In general, first-gen vendors have two primary concerns over next-gen marketing methods – both of which would be satisfied by joining AMTSO, and by listing on VirusTotal and abiding by its rules. These are a continued attempt to define first-gen vendors as solely signature-based technology purveyors; and a disinclination to submit to independent third-party comparative testing.

Signature-based

For this article, Cylance’s Skipper said that Cylance is a “significantly different technology than that of commodity signature based anti-virus or first-gen.” Cylance would be better served by demonstrating how its technology is significantly different to that of first-gen vendors (who also use machine learning and behavioral analysis) rather than implying that they are simply signature-based technologies.

SentinelOne’s Gainey said, “In the case of SentinelOne we use machine learning based analysis to detect malware embedded in binary images with extreme precision, which is the next evolution beyond the role that signatures have played for decades.” Again, the implication is that first-gen vendors are solely reliant on signatures where in reality they have employed machine learning for a lot longer than SentinelOne has existed.

Having said this, not all first-gen vendors consider next-gen vendors to be overly aggressive in their marketing. “I don’t see the approach of many new companies in the endpoint space that are focused on the modern threats as being particularly aggressive, per se,” commented Justin Dolly, CISO & CIO at Malwarebytes.

Third-party testing

Some, but not all, of the next-gen vendors simply claim that next-gen products cannot be fairly tested by third parties. With this argument, they decline to be tested. A recent blog from SentinelOne examines a malware sample that SentinelOne sometimes fails to detect. Its conclusion is that under some test conditions, the malware simply doesn’t activate; and without that activation, next-gen products will not detect it. “It’s easy to stuff a test set with malware samples which are either not valid executables or don’t behave maliciously and many tests are performed on freshly minted VM images with no user activity history, and running in the cloud which can be detected by interrogating IP address information.”

The implication is that it would be easy to create a test set that would drastically favor signature-based detection over next-gen behavioral detection. “A fair test must necessarily include current and functional samples executed in a realistic environment.”

But few people deny this final statement. The primary purpose of AMTSO is to develop testing standards that are fair to everyone. “The tester’s job,” Dennis Batchelder told SecurityWeek, “is to simulate as close to the real world as possible, and if a tester can do that he can measure how a customer would benefit from one product versus another product.”

Simon Edwards, director of independent test organization SELabs, accepts that this is his task; but doesn’t ultimately see a problem. Indeed, he doesn’t necessarily accept that first-gen and next-gen products cannot be directly compared.

“It makes sense,” Edwards told SecurityWeek, “for anti-malware users to be able to read useful test reports that investigate how effective all of the available products really are. I think that some of the newer companies have started to notice this demand and, now that their products are perhaps more mature, are feeling more inclined to enter tests willingly. I don’t think they will appear in separate ‘next gen’ reports. They claim to prevent malware and so it’s logical to include their results alongside those for other anti-malware products.”

The Way Forward

There is no easy, nor yet unsurmountable, solution to the arguments between first- and next-gen endpoint security vendors. Marketing is one area. First-gen vendors were so successful with the epithet ‘anti-virus’ that the name and technology (signature-based detection) stuck – even though their technology has become much more. Next-gen vendors have latched onto this marketing weakness with such ferocity that first-gens are as likely to develop new products that are based centrally around machine learning as they are to remarket their existing products. This occurred recently with both Symantec and Sophos, although the latter avoids ‘machine learning’ and just pitches itself as ‘next-generation’.

It would be naïve to expect all next-gen vendors to relax their marketing methods – not all are in it for the long haul. It is likely that some were conceived with a profitable exit already in mind. That would most likely be acquisition by a large company (possibly even an existing first-gen vendor). For this to happen they need rapid visibility and market share; and this is best gained in the short term by aggressive marketing.

Not all next-gen companies are like this. “Do testing companies know how to evaluate these next gen approaches? I think they are learning. It’s clear that the traditional way of testing is not optimal for the next gen companies but on the other hand it’s a poor excuse for next gen companies not to get tested,” said Invincea’s Ghosh. His solution to the difficulties in testing next-gen products is not to reject testing, but to join AMTSO and influence the test standards. Now he’s “helping to draft the standards around how you test next-gen products and technologies; there is a science to this and it shouldn’t be done without regard to scientific methods.”

So far only four next-gen companies have taken a similar route. But it’s a beginning. It may not yet be the beginning of the end of the endpoint wars; but it is the end of the beginning. Simon Edwards comments, “I would be surprised if there will be many well-known, credible anti-malware vendors not somehow involved with AMTSO or VirusTotal by next year.”

Related: Inside The Competitive Testing Battlefield of Endpoint Security