Security Experts:

Connect with us

Hi, what are you looking for?


Network Security

Palo Alto Networks, NSS Labs Spar Over NGFW Test Results

Next Generation Firewall Lab Test Sparks War of Words Over Testing Methodology

The war of words escalated this week between Palo Alto Networks and independent testing firm NSS Labs over the latest group test on next-generation firewalls.

Next Generation Firewall Lab Test Sparks War of Words Over Testing Methodology

The war of words escalated this week between Palo Alto Networks and independent testing firm NSS Labs over the latest group test on next-generation firewalls.

In the latest group test by NSS Labs, Palo Alto Networks was the only one of 12 vendors to receive a “caution” rating, a surprising drop from the “recommended” rating it received in the February 2013 test. NSS Labs said Palo Alto’s next-generation firewall “misses several critical evasions that leave its customers at risk” if run with the default configuration as attackers would be able to bypass the firewall’s detection capabilities.

The results prompted Palo Alto Networks’ senior vice president Lee Klarich to criticize NSS testing methodology in a blog post, noting the company decided to not participate in the test, which meant its firewall had not been optimally configured for the testbed. The NSS results did not make sense, since Palo Alto had invested heavily in the next-generation firewall’s security capabilities and updated the box at least twice since the last report. NSS raised issues that “have never been observed in other tests conducted internally or with our install base of over 19,000 global enterprises,” Kalrich said.

“The reason we did not participate in this test is that over time we have come to believe that the NSS model of allowing vendor test tuning prior to public test is a ‘pay-to-play’ approach and produces questionable objectivity and accuracy in results,” Klarich wrote.

NSS CEO Bob Walder responded to Kalrich in a blog post of his own, titled, “Seriously?”

“Palo Alto Networks was treated exactly the same as every other vendor in this test. NSS tests all NGFW products with the predefined vendor-recommended settings,” Walder said, dismissing Kalrich’s claim that the firewall wasn’t properly set up for the test. “The entire test is done on our dime, and the only input we ask from vendors is support in terms of supplying the most appropriate device, along with engineering support before and during the test, should we need it.”

The next-generation firewall appliances from Barracuda, Check Point Software Technologies, Cisco Systems, Cyberroam, Dell, Fortinet, Intel Security, WatchGuard, and Palo Alto Networks were part of the latest group. Appliances from Cisco and Cisco-Sourcefire scored the highest on security effectiveness, followed by Dell SonicWall and WatchGuard, according to the report. The tests depend on pre-defined vendor recommended settings because most customers deploy next-generation firewalls with the default configuration out of the box, Walder said.

“To reiterate, no tuning is permitted,” he said.

Klarich touted the Palo Alto’s research efforts and security investment over the past year, such as the team’s contributions to discovering vulnerabilities in Microsoft products and the company’s responses to Heartbleed and Shellshock. NSS findings didn’t match the real-world experience of their customers, especially since the customers do their “own stringent and detailed testing of our products in their mission critical environments,” Klarich said.

The latest round of controversy over testing comes after NSS said FireEye’s breach detection system did not work as well as competing products from Cisco and Trend Micro back in April. FireEye shot back with its own accusations of improper testing. NSS responded promptly with a blog post explaining its methodology adding more fuel to the fire. FireEye’s share price suffered shortly after the NSS report came out. While Palo Alto’s share price fell a bit after the report, it has recovered momentum fairly nicely. At the time of publishing on Oct. 3, shares of Palo Networks (NYSE:PANW) were trading up over 7 pecent, topping $105 per share.

Written By

Click to comment

Expert Insights

Related Content


Websites of German airports, administration bodies and banks were hit by DDoS attacks attributed to Russian hacker group Killnet

Network Security

NSA publishes guidance to help system administrators identify and mitigate cyber risks associated with transitioning to IPv6.

Identity & Access

Hackers rarely hack in anymore. They log in using stolen, weak, default, or otherwise compromised credentials. That’s why it’s so critical to break the...

Cybersecurity Funding

Forward Networks, a company that provides network security and reliability solutions, has raised $50 million from several investors.

Application Security

Electric car maker Tesla is using the annual Pwn2Own hacker contest to incentivize security researchers to showcase complex exploit chains that can lead to...

Network Security

Cisco patched a high-severity SQL injection vulnerability in Unified Communications Manager (CM) and Unified Communications Manager Session Management Edition (CM SME).

Cybersecurity Funding

Network security provider Corsa Security last week announced that it has raised $10 million from Roadmap Capital. To date, the company has raised $50...

Network Security

Vulnerabilities identified in TP-Link and NetComm router models could be exploited to achieve remote code execution (RCE).