Security Experts:

Connect with us

Hi, what are you looking for?

SecurityWeekSecurityWeek

Malware & Threats

Sophos Blasts Cylance’s Competitive Testing Methods

Sophos Says Cylance Rigged Demo Test

Sophos Says Cylance Rigged Demo Test

Dan Schiappa, VP with Sophos, has published what amounts to a stinging rebuke against Cylance product comparison methods. The specific incident involved a comparative test between Cylance and others, including Sophos. Cylance came out best — but a Sophos customer in the audience reportedly asked to see how the Sophos product was configured. He found that the default settings had been disabled. When enabled, and the tests re-run, Sophos was said to have beat Cylance.

Sophos does not use the word ‘cheating’. Nevertheless, suggestions of unfair environment manipulation are not new. What is unusual here is that Sophos is effectively and publicly inviting Cylance to join an open debate — or even to take legal action against Sophos.

“When the playing field is leveled, and Cylance’s product comes under real scrutiny,” wrote Schiappa, “the company cries foul, puts the fear of lawsuits into the minds of its partners, and accuses others of ‘smoke and mirrors’ tactics.”

Such behavior, which we could euphemistically describe as ‘gaming the opportunities’, are not new. As long ago as 1993 Sarah Tanner (probably better known as Alan Solomon) wrote ‘A Reader’s Guide to Reviews’ . Its purpose was to explain how tests can be manipulated, and why readers should not automatically believe everything they read. 

“The main weapons at your disposal,” says the article, “are the choice of what features to review and what to ignore and the weights given to the features you do cover. By a careful use of this, even GrottyScan can be the Editor’s Choice.” It then lists 26 different methods that can be used to tilt a comparison towards the tester’s own product. For example, if your own product is slow, “Do your timing test on a disk full of viruses. That way, WonderScan will be slowed down by the screen display and other things it has to do when it finds a virus, whereas GrottyScan won’t be slowed down, as it won’t have found many viruses.”

Much has changed in the intervening 23 years. Product testing and comparisons are far more sophisticated — but so is the ‘gaming’. In April 2015 Qihoo was accused of supplying units ‘cranked up’ to achieve well in speed tests, while consumers received slower but more reliable models. “On requesting an explanation from Qihoo 360 for their actions,” reported AV-Comparatives, AV-TEST and Virus Bulletin, “the firm confirmed that some settings had been adjusted for testing, including enabling detection of types of files such as keygens and cracked software, and directing cloud lookups to servers located closer to the test labs. After several requests for specific information on the use of third-party engines, it was eventually confirmed that the engine configuration submitted for testing differed from that available by default to users.”

The basic problem is that comparative product testing is very difficult, and very expensive. It is easy, either by intent or accident, to introduce bias in favor of one product over another. These difficulties persuaded the AV industry to form the anti-malware testing standards organization (AMTSO) to develop and promulgate transparent and accurate testing procedures.

AMTSO has its critics. It can easily be seen as an organization of the AV industry for the AV industry. But the work it has done is solid. Most of the major testing laboratories conform to or follow AMTSO standards — and it has the added advantage that no two members of AMTSO are likely to accuse each other of gaming tests undertaken to AMTSO standards. If gaming happens, as in the Qihoo incident, it will be discovered and made public.

As malware expert David Harley commented in an email to SecurityWeek, “If the tester is actually the vendor, there’s obvious scope for abuse. Vendors are and should be engaged in internal comparative testing, but I’d encourage the public to be skeptical where a vendor makes those results public. Not to assume malfeasance, but to examine claims and methodology to the best of their ability.”

The moral from both the Sophos experience with Cylance and the whole history of gaming test results is that the public should be aware of vendors’ own tests, and place more faith in independent tests operated under AMTSO overview.

Related: VirusTotal Policy Change Rocks Anti-Malware Industry

Related:  Palo Alto Networks, NSS Labs Spar Over NGFW Test Results

Written By

Kevin Townsend is a Senior Contributor at SecurityWeek. He has been writing about high tech issues since before the birth of Microsoft. For the last 15 years he has specialized in information security; and has had many thousands of articles published in dozens of different magazines – from The Times and the Financial Times to current and long-gone computer magazines.

Click to comment

Expert Insights

Related Content

Cyberwarfare

Websites of German airports, administration bodies and banks were hit by DDoS attacks attributed to Russian hacker group Killnet

Network Security

NSA publishes guidance to help system administrators identify and mitigate cyber risks associated with transitioning to IPv6.

Malware & Threats

Microsoft plans to improve the protection of Office users by blocking XLL add-ins from the internet.

Identity & Access

Hackers rarely hack in anymore. They log in using stolen, weak, default, or otherwise compromised credentials. That’s why it’s so critical to break the...

Cybercrime

CISA, NSA, and MS-ISAC issued an alert on the malicious use of RMM software to steal money from bank accounts.

Cybersecurity Funding

Forward Networks, a company that provides network security and reliability solutions, has raised $50 million from several investors.

Cybercrime

A recently disclosed vBulletin vulnerability, which had a zero-day status for roughly two days last week, was exploited in a hacker attack targeting the...

Cybercrime

Chinese threat actor DragonSpark has been using the SparkRAT open source backdoor in attacks targeting East Asian organizations.