Security Experts:

Connect with us

Hi, what are you looking for?


Network Security

Palo Alto Networks, NSS Labs Spar Over NGFW Test Results

Next Generation Firewall Lab Test Sparks War of Words Over Testing Methodology

The war of words escalated this week between Palo Alto Networks and independent testing firm NSS Labs over the latest group test on next-generation firewalls.

Next Generation Firewall Lab Test Sparks War of Words Over Testing Methodology

The war of words escalated this week between Palo Alto Networks and independent testing firm NSS Labs over the latest group test on next-generation firewalls.

In the latest group test by NSS Labs, Palo Alto Networks was the only one of 12 vendors to receive a “caution” rating, a surprising drop from the “recommended” rating it received in the February 2013 test. NSS Labs said Palo Alto’s next-generation firewall “misses several critical evasions that leave its customers at risk” if run with the default configuration as attackers would be able to bypass the firewall’s detection capabilities.

The results prompted Palo Alto Networks’ senior vice president Lee Klarich to criticize NSS testing methodology in a blog post, noting the company decided to not participate in the test, which meant its firewall had not been optimally configured for the testbed. The NSS results did not make sense, since Palo Alto had invested heavily in the next-generation firewall’s security capabilities and updated the box at least twice since the last report. NSS raised issues that “have never been observed in other tests conducted internally or with our install base of over 19,000 global enterprises,” Kalrich said.

“The reason we did not participate in this test is that over time we have come to believe that the NSS model of allowing vendor test tuning prior to public test is a ‘pay-to-play’ approach and produces questionable objectivity and accuracy in results,” Klarich wrote.

NSS CEO Bob Walder responded to Kalrich in a blog post of his own, titled, “Seriously?”

“Palo Alto Networks was treated exactly the same as every other vendor in this test. NSS tests all NGFW products with the predefined vendor-recommended settings,” Walder said, dismissing Kalrich’s claim that the firewall wasn’t properly set up for the test. “The entire test is done on our dime, and the only input we ask from vendors is support in terms of supplying the most appropriate device, along with engineering support before and during the test, should we need it.”

The next-generation firewall appliances from Barracuda, Check Point Software Technologies, Cisco Systems, Cyberroam, Dell, Fortinet, Intel Security, WatchGuard, and Palo Alto Networks were part of the latest group. Appliances from Cisco and Cisco-Sourcefire scored the highest on security effectiveness, followed by Dell SonicWall and WatchGuard, according to the report. The tests depend on pre-defined vendor recommended settings because most customers deploy next-generation firewalls with the default configuration out of the box, Walder said.

“To reiterate, no tuning is permitted,” he said.

Klarich touted the Palo Alto’s research efforts and security investment over the past year, such as the team’s contributions to discovering vulnerabilities in Microsoft products and the company’s responses to Heartbleed and Shellshock. NSS findings didn’t match the real-world experience of their customers, especially since the customers do their “own stringent and detailed testing of our products in their mission critical environments,” Klarich said.

The latest round of controversy over testing comes after NSS said FireEye’s breach detection system did not work as well as competing products from Cisco and Trend Micro back in April. FireEye shot back with its own accusations of improper testing. NSS responded promptly with a blog post explaining its methodology adding more fuel to the fire. FireEye’s share price suffered shortly after the NSS report came out. While Palo Alto’s share price fell a bit after the report, it has recovered momentum fairly nicely. At the time of publishing on Oct. 3, shares of Palo Networks (NYSE:PANW) were trading up over 7 pecent, topping $105 per share.

Written By

Click to comment

Daily Briefing Newsletter

Subscribe to the SecurityWeek Email Briefing to stay informed on the latest threats, trends, and technology, along with insightful columns from industry experts.

Join this webinar to learn best practices that organizations can use to improve both their resilience to new threats and their response times to incidents.


Join this live webinar as we explore the potential security threats that can arise when third parties are granted access to a sensitive data or systems.


Expert Insights

Related Content

Identity & Access

Zero trust is not a replacement for identity and access management (IAM), but is the extension of IAM principles from people to everyone and...

Network Security

NSA publishes guidance to help system administrators identify and mitigate cyber risks associated with transitioning to IPv6.


Websites of German airports, administration bodies and banks were hit by DDoS attacks attributed to Russian hacker group Killnet

Identity & Access

Hackers rarely hack in anymore. They log in using stolen, weak, default, or otherwise compromised credentials. That’s why it’s so critical to break the...

Network Security

Attack surface management is nothing short of a complete methodology for providing effective cybersecurity. It doesn’t seek to protect everything, but concentrates on areas...

Cybersecurity Funding

Network security provider Corsa Security last week announced that it has raised $10 million from Roadmap Capital. To date, the company has raised $50...

Network Security

Our networks have become atomized which, for starters, means they’re highly dispersed. Not just in terms of the infrastructure – legacy, on-premises, hybrid, multi-cloud,...

Application Security

Fortinet on Monday issued an emergency patch to cover a severe vulnerability in its FortiOS SSL-VPN product, warning that hackers have already exploited the...