CrowdStrike Vs NSS Labs, Round 2: NSS Hits Back

In February 2017, endpoint protection firm CrowdStrike took the unusual step of suing independent product testing organization NSS Labs, “to hold it accountable for unlawfully accessing our software, breaching our contract, pirating our software, and improper security testing.”

The immediate purpose of the suit was to support action for an injunction to prevent NSS Labs from publishing test result details of CrowdStrike’s Falcon endpoint security product within its latest public test. The injunction failed, and NSS published the results.

At the time, NSS Labs issued brief statements but published no lengthy response to CrowdStrike’s blogged accusations of ‘unlawful conduct’ and ‘deeply flawed methodology’. Now it has done so.

“Given the serious inaccuracies CrowdStrike has been promoting in their blog and elsewhere, we decided that we needed to tell our side of the story,” blogged NSS CEO Vikram Phatak. The blog amounts to a step-by-step refutation of CrowdStrike’s accusations.

Where CrowdStrike claims the tests are incomplete (it disconnected its cloud-based Falcon before the tests were complete) and the results therefore invalid, NSS claims that CrowdStrike’s results were not penalized. “CrowdStrike did not receive a zero (0) for the parts of the test we were unable to complete – because we believed that penalizing CrowdStrike for disabling the product could mislead the public.” It also points out that Falcon had missed various attacks before the disconnection, and that those attacks would remain missed whether the full testing had been completed or not.

A primary thrust of CrowdStrike’s arguments is that it had “declined to participate in a public test after completing a private test with NSS, based on NSS’ flawed and improper testing execution.”

The NSS response is that it is not open for individual companies to withdraw from a public test. “NSS Labs informed CrowdStrike that our position, as always, is that if a product is good enough to sell to the public, it is good enough to be tested and that we would purchase their product if necessary.” NSS tried to buy the product, was blocked by CrowdStrike, but “found an enterprise who would be willing to work with us to purchase the product.”

CrowdStrike Falcon was subsequently part of the NSS public tests, but failed to complete because CrowdStrike disconnected it from its cloud before completion.

It is an unsightly squabble; but one that has been threatening for many months. Next-gen endpoint protection firms have tended to claim that the in situ anti-virus products do not work. Those ‘legacy’ firms have responded that independent testing would settle the issue. To begin with, next-gens replied that their products could not be tested in the same way as legacy products (and it should be said that they had a point).

The testing laboratories, however, have spent considerable time and effort in improving their testing techniques specifically for next gens — and many next-gens are now happy to take part. Three other next-gen products included in the same tests did rather well: Cylance at 99.69%, SentinelOne at 99.79%, and Invincea at 99.49%. CrowdStrike did less well at 74.17%.

Anup Ghosh, founder and CEO at Invincea, accepts that there have been difficulties in testing, but believes that cooperation rather than withdrawal is the answer. “We are really excited about how well we did in the NSS Labs AEP test,” he told SecurityWeek. “We won’t comment on competitors or competitors’ behavior. I think you know our stance on third party testing: it should be done early and often and with multiple reputable third party testers. NSS Labs does a good job in ‘real world’ exploits and evasions techniques, but every test shop has its pros and cons. That’s why we try to participate in as many public reputable third party tests as possible.”

In response to a SecurityWeek inquiry, CrowdStrike provided the following statement on Tuesday, March 7:

“We are aware that NSS Labs, a pay-for-play, for-profit business, published a blog on March 2 relating to the legal action we initiated against them on February 10 and the blogs we published the week of February 12.  As they state at the end of the third paragraph of their recent post, ‘Unfortunately, nothing has changed.’

As we blogged on February 15, ‘Taken in total, NSS’ failure to conduct the most basic of fact checking during the private testing and the well-publicized history of problems with NSS testing ultimately gave us no confidence that NSS Labs could conduct accurate testing of our security products. Therefore, we declined to participate in the public test.’

While we continue to pursue the legal process focused on unlawfully accessing our software and a subsequent incomplete and materially flawed test, we encourage all security users to continue to rely on the independent testing and certification results of reputable labs like AV-Comparatives and SE Labs who have independently tested and certified CrowdStrike Falcon has been as an effective AV replacement. We would also direct users to look to the guidance of Gartner, who recently named CrowdStrike a Visionary in the Gartner Magic Quadrant for Endpoint Protection Platforms.”