WatchGuard Falls To NGFW Test, Questions Outcome

Executives at security appliance maker WatchGuard Technologies said the firm is accepting at least some of the results of a battery of extensive tests on its next-generation firewall that ultimately caused the appliance to fall well below its competitors.

NSS Labs tested nine NGFW appliances provided by Check Point, Dell-SonicWall, Fortinet, Juniper Networks, Palo Alto Networks, Sourcefire, Stonesoft and WatchGuard. The results of the study were issued by the firm last week.

In tests conducted on the WatchGuard XTM 2050 appliance, the company was able to detect roughly half of the evasion techniques the testing team pitted against it. The widely known hacker techniques, such as HTML obfuscation, help disguise an attack to avoid being detected by the security appliance and is weighted heavily in the NSS Labs' tests.

"Although WatchGuard demonstrated a good level of exploit protection, it was let down by its poor antievasion capabilities and a suboptimal price-performance ratio," NSS Labs noted in its report.

[Related: 9 Unified Threat Management Security Appliances To Watch In 2013 ]

Frank Artes, director of research at NSS Labs, said the company also had some shortcomings to its system management capabilities. The company's system manager was not as scalable and mature as other vendor products. WatchGuard was the only vendor tested by NSS Labs that was given a "caution" designation.

"It really hurts them when other aspects that they should be doing aren't scoring at 100 percent," Artes said. "We expect WatchGuard, if it puts in the tweaks necessary, between now and next year it could be sitting shoulder to shoulder with Check Point."

WatchGuard also failed to integrate with NSS Labs' Active Directory implementation, making it unable to correctly enforce user-based policies. The appliance also scored lower than its peers in blocking remote attacks against vulnerable applications, blocking about 85 percent of the remote attacks. It had an overall rate of blocking of 91 percent of exploits.

Dave Taylor, vice president of corporate strategy at Seattle-based WatchGuard, said the technology is tuned as a unified threat management appliance and not as a firewall. The firm's appliance was tested by NSS Labs in the firewall category last year and the overall results came out better.

"It absolutely is not an accurate representation of what our customers will find. Our customers that run our product love it, and we are perceived very highly in the market," Taylor said. "We don't optimize for NGFW; we optimize for UTM." Taylor said the results were "frustrating," explaining to CRN that while other vendor NGFW appliances do single-pass testing, WatchGuard uses seven technologies to scan network packets for malware and anomalous activity. The firm's development team continues to work on making the appliance easy to deploy and maintain, and technology partners are constantly boosting their effectiveness, Taylor said. WatchGuard also has failover and system redundancy capabilities, he said.

NEXT: Corrupted Firmware Caused Delay

During two days of tests, Taylor said the firm faced a considerable challenge when it was forced to download and reinstall new firmware because the original firmware was corrupted. The installation process took 16 hours, he said. NSS Labs said six of the vendors it tested submitted products that required firmware updates or configuration changes to complete the NSS tests.

The WatchGuard firmware setback gave engineers little time to connect the appliance to NSS Labs' Active Directory implementation, Taylor said. The company has supported AD for five years, he said. "We ran out of time before we had enough opportunity to diagnose it," Taylor said. "AD integration is not only important, it is critical in this space. We think the problem came from their implementation of AD."

Taylor said the data in WatchGuard's individual Product Analysis Report (PAR) compiled by NSS Labs varies significantly to the testing firms' comparative analysis. The calculations indicate that there is a significant amount of subjectivity in the test results, he said.

NSS Labs' Artes said the firm stands by its testing procedures and reports. "There's no hidden manipulation of the scores," Artes said. "They got 50 percent for evasions and they're expected to get 100 percent, and that had a tremendous impact on their overall score."

All of the other vendors tested by NSS Labs came out above average in the testing firms' Security Value Map, with Check Point's 12600 appliance getting a 98.5 percent security effectiveness rating. The Sourcefire 8250 and 8290 appliances also fared above average, followed by Dell-SonicWall SuperMassive E10800, Fortinet FortiGate 3600C, Stonesoft 3202 and Palo Alto Networks PA-5020. Juniper Networks' SRX 3600 got above-average scores for protection and management but earned a neutral rating as a result of NSS Labs' Value calculation.

While many of the vendors had application control, a key component in an NGFW, the Palo Alto appliance stood out for its ability to drill down into an application. Through a checkbox interface, a networking pro can easily block features in certain applications from running, said NSS Labs' Artes. "Their entire approach for administration of an NGFW is from an application standpoint not the layer standpoint," Artes said. "It's a great delineation and interesting, new approach."

Other appliances provide application control through protocol and port utilization, a more complicated process that requires an understanding of how the application works, Artes said.

PUBLISHED MARCH 6, 2013