Palo Alto Networks Earns Poor Results, Caution Rating In NGFW Test

Palo Alto Networks' next-generation firewall earned a "caution" rating from NSS Labs following the testing firm's recent bake-off, in which it failed to protect against certain attacks and underperformed against competitor appliances.

Palo Alto Networks, which was among the first to be designated a "next generation" firewall for its application control and other security components within its line of appliances, fell short against Check Point Software Technologies, Cisco Systems, Dell, Fortinet, Intel Security (formerly McAfee) and WatchGuard.

NSS Labs said the Palo Alto PA-3020 Appliance passed stability and reliability tests, and enforced firewall policies. It also correctly enforced complex outbound and inbound policies, the firm said. The appliance fell short in detecting evasion measures often used by attackers to bypass firewalls. Using RPC and IP Fragmentation attacks, NSS Labs was able to conduct a bypass. The appliance also took a performance hit, earning a 719-Mbps rating while the vendor claimed 1-Gbps performance.

[Related: Palo Alto Networks, FireEye Criticize NSS Labs; Testing Firm Defends Itself]

The Palo Alto testing gave the company's appliance a 60.9 percent security effectiveness score and a below-average total-cost-of-ownership rating. The closest vendor was Cyberoam, which got an 88.2 percent security effectiveness score and an above-average total-cost-of-ownership rating, earning it a neutral designation from NSS Labs.

A Palo Alto Networks spokesperson declined to comment about the test results but referred CRN to a blog post rebuking the NSS Labs test results. In an interview with CRN earlier this year, Nir Zuk, the company's chief technology officer, was critical of NSS Labs, calling the firm's testing methodology "questionable."

In response to the latest round of testing, Lee Klarich, senior vice president of product management at Palo Alto Networks, said the company did not cooperate in the NSS Labs tests. The poor scores may have resulted because, unlike competitor products in the tests, Palo Alto did not provide guidance on the configuration and tuning of the device.

Klarich also said the test results are questionable because of strong performance scores the company received in NSS Labs testing conducted in 2013. Palo Alto also did not participate in that test, he said.

"It is also interesting to note that they say that we updated our OS in that time and broke the technology," Klarich said. "There is no basis for that claim as best evidenced by the fact that, in the last year alone, we added almost 6,000 new customers, all of whom have done their own stringent and detailed testing of our products in their mission-critical environments."

Palo Alto Networks is only a couple of weeks off of its annual partner conference in Chicago, where the vendor touted its strong sales against appliances from Check Point, Cisco Systems, Fortinet and McAfee.

Testing results are part of the evaluation process, but prospective clients need to consider a variety of other factors, including the usability of the management console, reporting capabilities and the ability to drill down and investigate specific threats, said J.D. Butt, vice president of solutions at Chicago-based solution provider Nexum, a Palo Alto Networks partner that also specializes in selling and deploying a variety of networking security vendor appliances.

"Organizations need to look and read in between the lines of some of these tests," Butt said. "No appliance is able to protect as well in a default configuration, and no one rolls out a default configuration."

Butt said performance is paramount to most discussions with prospective clients. Customers want an appliance to meet their needs for the next three to five years, he said.

"It is really all about how a device is configured, and that goes into how effective it is," Butt said. "The wrong person configuring it, or architecting a solution, makes the client vulnerable."

Security vendors have lashed out at NSS Labs testing in the past. FireEye received substandard marks against competitors in a bake-off earlier this year that didn't include Palo Alto Networks. In 2013, WatchGuard's XTM 2050 appliance got poor marks and earned a caution designation after it fell well below competitors.

Sales teams reference test results often to poke holes against competitors, according to solution providers. Often, poor test results are pushed heavy-handedly, said Scott Fuhriman, a network security pro and vice president of sales and product development at TierPoint. Fuhriman said customers need to conduct their own evaluation and choose the best appliance that meets their requirements.

"If you test products a certain way, you can create different results that are favorable for one vendor or another regardless of how independent the test is," Fuhriman said. "You get the best results when an appliance is properly configured, deployed, and routinely maintained and monitored."

PUBLISHED SEPT. 30, 2014