Palo Alto Networks, FireEye Criticize NSS Labs; Testing Firm Defends Itself

Network security industry rivals Palo Alto Networks and FireEye are questioning the credibility of tests conducted by NSS Labs, following a new report that aimed to test the effectiveness and gauge the total cost of ownership of the industry's so-called breach detection appliances.

FireEye had declined to participate in the recent testing, but engineers at NSS Labs proceeded with tests on the company's Malware Protection System (MPS) appliance anyway. It scored "below average" in the NSS Labs comparative group product test. Palo Alto Networks was not invited to participate in the testing firm's latest report.

In a wide-ranging interview with CRN, Palo Alto Networks CTO Nir Zuk criticized the legitimacy of NSS Labs' testing methodology. The security networking vendor has declined to participate in NSS Labs testing in the past because of what Zuk calls a "flawed sales model."

[Related: NSS Labs Intrusion Prevention Tests: Did Your Vendor Partner Pass? ]

In order to get as many vendors in the report as possible, according to Zuk, the company sets the testing methodology very low, enabling "mediocre vendors" to compete. The company issues licenses called "reprint rights" to vendors for publishing testing results, negotiating a fee with vendors that can exceed $100,000 before the testing results are made public. The fee gives vendors the right to publicize the report and distribute it to potential clients.

"You don't want to do a report on two vendors; you want to do a report on 10 vendors and charge each of them $100,000, and that's how you make money," Zuk said. "If you are going to have a very high bar for your test and only one or two vendors are going to succeed in your test, the vendors are going to stop paying for it."

NSS Labs can't generate enough revenue from customers so it sets the bar where the vendors want it, Zuk said.

"The vendors pay a lot of money; this is all vendor-paid and there's a degree of influence," Zuk said. "If you think you are much, much better than others and you think the bar should be much higher, then maybe you don't want to participate in some of these tests."

The report in question, "Breach Detection Systems Comparative Analysis and Security Value Map," was issued April 2 by NSS Labs. It evaluated security appliances from AhnLab, Fidelis, FireEye, Fortinet, Sourcefire (Cisco) and Trend Micro, establishing metrics on a wide range of issues, from the time it takes to deploy and configure the devices to the effectiveness in detecting malware and exploits used by attackers.

NEXT: NSS Labs Responds To Criticism

FireEye CTO Dave Merkel first issued a statement Wednesday questioning the legitimacy of the report. FireEye, which has been gaining much attention following its successful IPO last December, has been highlighting the number of zero-day threats it detected in 2013 as part of its marketing strategy.

"Any lab test is fundamentally unable to replicate the targeted, advanced attacks launched by sophisticated criminal networks and nation-states," Merkel said in the statement.

In a detailed blog post, FireEye's senior vice president of products, Manish Gupta, called the NSS methodology "severely flawed." Gupta said FireEye no longer participates in the tests, insisting that they should run in a production environment. .

"The FireEye product they used was not even fully functional, leveraged an old version of our software and didn’t have access to our threat intelligence [unlike our customers]," Gupta said.

The FireEye appliance also wasn't connected to the FireEye threat intelligence feed to receive blacklisting updates. Gupta criticized the malware samples used during the testing process. The FireEye appliance missed some common threats, but Gupta said all the vendor products should have been tested against "new and unknown" threats, such as a zero-day exploit.

NSS Labs is fully defending its methodology and is standing by the legitimacy of the results. Palo Alto Networks was going to be included in the next round of testing, said Vikram Phatak, who told CRN Wednesday that he was perplexed by the Palo Alto Networks criticism.

NSS Labs does not conduct a "pay-to-play" model of testing in which vendors must pay to participate, Phatak said, adding that the firm also got out of the certification business in 2009 to bolster the legitimacy of its tests. NSS Lab engineers conduct testing based on customer requests. If a vendor declines to participate in testing, NSS Labs will buy the appliance or software to conduct tests, he said.

"In any test and every test that is published and made public we do not take a single penny from the vendors that are examined," Phatak said, saying the firm attempts to mirror the Consumer Reports testing practices. "Most of the money we receive to pay our bills come from enterprise clients, many banks and oil companies, who require an evaluation based on testing data without the subjectivity."

Phatak addressed the FireEye criticism as well, saying that NSS Labs tested the appliance against live, real-time exploits and malware. The FireEye platform couldn't support detection on all platforms tested, resulting in subpar results.

"There are other products out there that are probably better based on what the data is saying," Phatak said. "The core issue is that they had a product that didn't work with 64-bit operating systems,"

The product also didn't get the highest detection rates because it uses an open-source antivirus engine (ClamAv), in addition to whitelists and other methods to detect common malware strains. The combined, standard detection methods are not 100 percent effective, Phatak said. To detect suspicious files, the vendor tests them in 10 virtual machine detonation chambers.

Solution providers reached by CRN declined to speak about the criticism against NSS Labs. They called the reports useful for some larger clients that conduct a systematic evaluation process. A thorough evaluation typically involves attempting to review tests from an independent source, according to security experts.

PUBLISHED APRIL 3, 2014