Connect with us

Hi, what are you looking for?


Endpoint Security

CrowdStrike Sues NSS Labs to Prevent Publication of Test Results

CrowdStrike filed suit against NSS Labs

CrowdStrike filed suit against NSS Labs

On February 10, 2017, next-gen endpoint protection firm CrowdStrike filed suit against security product testing firm NSS Labs, and sought a temporary restraining order to prevent publication of CrowdStrike comparative test results. On February 13, the injunction was denied by the District Court of Delaware. On February 14, NSS published the results as part of its Advanced Endpoint Protection Group Test Results.

CrowdStrike explained the background in a blog post yesterday. It filed suit, it said, to hold NSS “accountable for unlawfully accessing our software, breaching our contract, pirating our software, and improper security testing. Regardless of test results (which we have not seen), CrowdStrike is making a stand against what we believe to be unlawful conduct.”

CrowdStrike had earlier commissioned NSS to undertake a private test of its products, but was dissatisfied with the test methods, calling them “deeply flawed”. Because of this it decided not to participate in the subsequent public test, and prohibited NSS from using its software. But according to CrowdStrike, NSS “colluded with a reseller and engaged in a sham transaction to access our software to conduct the testing. In doing so, NSS breached their contract with CrowdStrike, violated our end user licensing agreement (EULA), misappropriated our intellectual property, and improperly used credentials. Once we became aware that an unauthorized user account associated with a reseller was used for the tests, we suspended access immediately. Any test results that NSS did obtain are incomplete and materially flawed.”

Product testing has long been a problem for the newer endpoint protection companies. In June 2016, Sophos blasted Cylance, and added, “when the playing field is leveled, and Cylance’s product comes under real scrutiny, the company cries foul, puts the fear of lawsuits into the minds of its partners, and accuses others of ‘smoke and mirrors’ tactics.”

Now the threat of a lawsuit has become a reality between CrowdStrike and NSS Labs. In the meantime, many of the new endpoint protection companies, including Cylance, have modified their attitudes. Cylance was amongst the tested products, as was SentinelOne and Invincea. These last three did rather well in the overall scores: Cylance at 99.69%, SentinelOne at 99.79%, and Invincea at 99.49%. CrowdStrike did less well at 74.17% — but as CrowdStrike claimed, the results ‘are incomplete’; and as NSS Labs admits, “The Falcon Host’s final rating may have been different had it completed the test.”

There are two primary issues here: is it possible to conduct fair comparative tests for advanced endpoint protection products (aka, machine-learning or next-gen AV); and is the law a valid method of preventing them?

Opinions differ on the first. David Harley blogged in WeLiveSecurity on Monday (although I understand it was written well before this current issue): Next-gen security software: ‘Myths and marketing’. Quoting a question I asked him months ago (basically, is there any way to compare 1st- and 2nd-gen AV products), he said, “yes, of course there is.”

Advertisement. Scroll to continue reading.

Vesselin Bontchev, who is possibly the ultimate culprit (“I practically invented independent competent anti-virus testing while I was working at the Virus Test Center at the University of Hamburg in the early ’90s”) takes the opposite view. He believes that neither products nor testing are anywhere near as competent as they should be. “Whenever there is a major conflict, like this CrowdStrike vs NSS Labs story,” he wrote yesterday, “you can usually bet that both sides are in the wrong. CrowdStrike probably have a crappy product they want to sell and didn’t like the test results, while NSS Labs probably has a crappy and/or incomplete testing methodology and CrowdStrike found some legitimate flaws in it.”

The law, however, is a heavy instrument to prevent public testing. SecurityWeek asked NSS to comment, and was told via email by CEO Vikram Phatak, “While CrowdStrike’s request for a Temporary Restraining Order and Preliminary Injunction were denied by the Federal court, they are still suing us at present, and so we are limited in what we can say. Whether or not it is their intent, their suit has the effect of keeping us from debating the facts publicly.  

“We obviously disagree and are disappointed with CrowdStrike’s characterization of NSS as portrayed in their recent blog post… And as far as Crowdstrike’s suit against NSS, we believe the judge’s ruling and memorandum speak for themselves.”

SecurityWeek also approached CrowdStrike, the Anti-Malware Testing Standards Organization (of which both CrowdStrike and NSS Labs are members), and another independent test lab for comments. We have so far received no response (although an informal reply from CrowdStrike did say, “Things are moving quickly today. Keep an eye on your inbox for an update”). If any comments are received they will be added as an update to this post.

Meanwhile, customers are left with an ongoing problem: can test results be trusted? There is no easy answer to this. The best solution is for customers to insist on an on-site trial periods to see whether a preferred solution is actually up to the job.

Written By

Kevin Townsend is a Senior Contributor at SecurityWeek. He has been writing about high tech issues since before the birth of Microsoft. For the last 15 years he has specialized in information security; and has had many thousands of articles published in dozens of different magazines – from The Times and the Financial Times to current and long-gone computer magazines.

Click to comment

Daily Briefing Newsletter

Subscribe to the SecurityWeek Email Briefing to stay informed on the latest threats, trends, and technology, along with insightful columns from industry experts.

SecurityWeek’s Threat Detection and Incident Response Summit brings together security practitioners from around the world to share war stories on breaches, APT attacks and threat intelligence.


Securityweek’s CISO Forum will address issues and challenges that are top of mind for today’s security leaders and what the future looks like as chief defenders of the enterprise.


Expert Insights

Related Content

Endpoint Security

Today, on January 10, 2023, Windows 7 Extended Security Updates (ESU) and Windows 8.1 have reached their end of support dates.

CISO Strategy

Varied viewpoints as related security concepts take on similar traits create substantial confusion among security teams trying to evaluate and purchase security technologies.

Application Security

Virtualization technology giant VMware on Tuesday shipped urgent updates to fix a trio of security problems in multiple software products, including a virtual machine...

Endpoint Security

The Zero Day Dilemma

Application Security

Password management firm LastPass says the hackers behind an August data breach stole a massive stash of customer data, including password vault data that...

Endpoint Security

When establishing visibility and security controls across endpoints, security professionals need to understand that each endpoint bears some or all responsibility for its own...

Application Security

Microsoft on Tuesday pushed a major Windows update to address a security feature bypass already exploited in global ransomware attacks.The operating system update, released...

Application Security

Electric car maker Tesla is using the annual Pwn2Own hacker contest to incentivize security researchers to showcase complex exploit chains that can lead to...