On February 10, 2017, next-gen endpoint protection firm CrowdStrike filed suit against security product testing firm NSS Labs, and sought a temporary restraining order to prevent publication of CrowdStrike comparative test results. On February 13, the injunction was denied by the District Court of Delaware. On February 14, NSS published the results as part of its Advanced Endpoint Protection Group Test Results.
CrowdStrike explained the background in a blog post yesterday. It filed suit, it said, to hold NSS “accountable for unlawfully accessing our software, breaching our contract, pirating our software, and improper security testing. Regardless of test results (which we have not seen), CrowdStrike is making a stand against what we believe to be unlawful conduct.”
CrowdStrike had earlier commissioned NSS to undertake a private test of its products, but was dissatisfied with the test methods, calling them “deeply flawed”. Because of this it decided not to participate in the subsequent public test, and prohibited NSS from using its software. But according to CrowdStrike, NSS “colluded with a reseller and engaged in a sham transaction to access our software to conduct the testing. In doing so, NSS breached their contract with CrowdStrike, violated our end user licensing agreement (EULA), misappropriated our intellectual property, and improperly used credentials. Once we became aware that an unauthorized user account associated with a reseller was used for the tests, we suspended access immediately. Any test results that NSS did obtain are incomplete and materially flawed.”
Product testing has long been a problem for the newer endpoint protection companies. In June 2016, Sophos blasted Cylance, and added, “when the playing field is leveled, and Cylance’s product comes under real scrutiny, the company cries foul, puts the fear of lawsuits into the minds of its partners, and accuses others of ‘smoke and mirrors’ tactics.”
Now the threat of a lawsuit has become a reality between CrowdStrike and NSS Labs. In the meantime, many of the new endpoint protection companies, including Cylance, have modified their attitudes. Cylance was amongst the tested products, as was SentinelOne and Invincea. These last three did rather well in the overall scores: Cylance at 99.69%, SentinelOne at 99.79%, and Invincea at 99.49%. CrowdStrike did less well at 74.17% — but as CrowdStrike claimed, the results ‘are incomplete’; and as NSS Labs admits, “The Falcon Host’s final rating may have been different had it completed the test.”
There are two primary issues here: is it possible to conduct fair comparative tests for advanced endpoint protection products (aka, machine-learning or next-gen AV); and is the law a valid method of preventing them?
Opinions differ on the first. David Harley blogged in WeLiveSecurity on Monday (although I understand it was written well before this current issue): Next-gen security software: ‘Myths and marketing’. Quoting a question I asked him months ago (basically, is there any way to compare 1st- and 2nd-gen AV products), he said, “yes, of course there is.”
Vesselin Bontchev, who is possibly the ultimate culprit (“I practically invented independent competent anti-virus testing while I was working at the Virus Test Center at the University of Hamburg in the early ’90s”) takes the opposite view. He believes that neither products nor testing are anywhere near as competent as they should be. “Whenever there is a major conflict, like this CrowdStrike vs NSS Labs story,” he wrote yesterday, “you can usually bet that both sides are in the wrong. CrowdStrike probably have a crappy product they want to sell and didn’t like the test results, while NSS Labs probably has a crappy and/or incomplete testing methodology and CrowdStrike found some legitimate flaws in it.”
The law, however, is a heavy instrument to prevent public testing. SecurityWeek asked NSS to comment, and was told via email by CEO Vikram Phatak, “While CrowdStrike’s request for a Temporary Restraining Order and Preliminary Injunction were denied by the Federal court, they are still suing us at present, and so we are limited in what we can say. Whether or not it is their intent, their suit has the effect of keeping us from debating the facts publicly.
“We obviously disagree and are disappointed with CrowdStrike’s characterization of NSS as portrayed in their recent blog post… And as far as Crowdstrike’s suit against NSS, we believe the judge’s ruling and memorandum speak for themselves.”
SecurityWeek also approached CrowdStrike, the Anti-Malware Testing Standards Organization (of which both CrowdStrike and NSS Labs are members), and another independent test lab for comments. We have so far received no response (although an informal reply from CrowdStrike did say, “Things are moving quickly today. Keep an eye on your inbox for an update”). If any comments are received they will be added as an update to this post.
Meanwhile, customers are left with an ongoing problem: can test results be trusted? There is no easy answer to this. The best solution is for customers to insist on an on-site trial periods to see whether a preferred solution is actually up to the job.