The war of words escalated this week between Palo Alto Networks and independent testing firm NSS Labs over the latest group test on next-generation firewalls.
In the latest group test by NSS Labs, Palo Alto Networks was the only one of 12 vendors to receive a “caution” rating, a surprising drop from the “recommended” rating it received in the February 2013 test. NSS Labs said Palo Alto’s next-generation firewall “misses several critical evasions that leave its customers at risk” if run with the default configuration as attackers would be able to bypass the firewall’s detection capabilities.
The results prompted Palo Alto Networks’ senior vice president Lee Klarich to criticize NSS testing methodology in a blog post, noting the company decided to not participate in the test, which meant its firewall had not been optimally configured for the testbed. The NSS results did not make sense, since Palo Alto had invested heavily in the next-generation firewall’s security capabilities and updated the box at least twice since the last report. NSS raised issues that “have never been observed in other tests conducted internally or with our install base of over 19,000 global enterprises,” Kalrich said.
“The reason we did not participate in this test is that over time we have come to believe that the NSS model of allowing vendor test tuning prior to public test is a ‘pay-to-play’ approach and produces questionable objectivity and accuracy in results,” Klarich wrote.
NSS CEO Bob Walder responded to Kalrich in a blog post of his own, titled, “Seriously?”
“Palo Alto Networks was treated exactly the same as every other vendor in this test. NSS tests all NGFW products with the predefined vendor-recommended settings,” Walder said, dismissing Kalrich’s claim that the firewall wasn’t properly set up for the test. “The entire test is done on our dime, and the only input we ask from vendors is support in terms of supplying the most appropriate device, along with engineering support before and during the test, should we need it.”
The next-generation firewall appliances from Barracuda, Check Point Software Technologies, Cisco Systems, Cyberroam, Dell, Fortinet, Intel Security, WatchGuard, and Palo Alto Networks were part of the latest group. Appliances from Cisco and Cisco-Sourcefire scored the highest on security effectiveness, followed by Dell SonicWall and WatchGuard, according to the report. The tests depend on pre-defined vendor recommended settings because most customers deploy next-generation firewalls with the default configuration out of the box, Walder said.
“To reiterate, no tuning is permitted,” he said.
Klarich touted the Palo Alto’s research efforts and security investment over the past year, such as the team’s contributions to discovering vulnerabilities in Microsoft products and the company’s responses to Heartbleed and Shellshock. NSS findings didn’t match the real-world experience of their customers, especially since the customers do their “own stringent and detailed testing of our products in their mission critical environments,” Klarich said.
The latest round of controversy over testing comes after NSS said FireEye’s breach detection system did not work as well as competing products from Cisco and Trend Micro back in April. FireEye shot back with its own accusations of improper testing. NSS responded promptly with a blog post explaining its methodology adding more fuel to the fire. FireEye’s share price suffered shortly after the NSS report came out. While Palo Alto’s share price fell a bit after the report, it has recovered momentum fairly nicely. At the time of publishing on Oct. 3, shares of Palo Networks (NYSE:PANW) were trading up over 7 pecent, topping $105 per share.