Symantec's Certificate Authority 'Vault': $11M Worth Of James Bond-Like Security

Designed to be a virtual showcase for advanced security, Symantec's certificate authority vault boasts a collection of some of the most advanced security measures aimed at protecting digital certificates that power the e-commerce industry today. Before CRN's tour of the "vault" even began, the specter of extremely tight security became clear.

When the media relations rep pulled his employee badge from the retractable reel attached to his belt and held the card to the electronic reader mounted at the entrance of the facility, nothing happened. Anywhere else on the Symantec campus, that badge would've worked like a charm. But, this nondescript, unmarked building was different. It houses much of Symantec's most sensitive and sophisticated infrastructure around PKI and digital certificates. The value of the information stored here is limited only by the imagination of the crook who might try to steal some of it. That has never happened in the past, and the objective of this tour was to show CRN how and why it hasn't.

The voice of a security guard crackled through the intercom and grilled the media rep as to who we were and why we were there. We were eventually buzzed into a small lobby with four interior doors; three of which had electronic locks that required both a card key and a PIN, similar to the exterior door. The fourth was an ordinary conference room occupied by people who would deliver the initial briefing.

[Related: RSA Fraud Report: Security By the Numbers ]

A small tray of food waited on a side table. We silently wondered if those sandwiches might have tiny tracking devices that would transmit our whereabouts for the rest of our lives. What we were about to see was seemingly inspired by Tom Clancy novels, but the sandwiches had to be low-tech.

We're pretty sure they were, anyway.

From the time that e-commerce first became popular, its continued success has largely been based on digital certificates that are designed to establish that the buyer is genuinely sending payment information to the company with whom they are actually trying to do business. Anyone who can get between the buyer and the seller would have a wealth of information that could most literally be converted into a wealth of wealth.

NEXT: Security At A Higher Level

The development of the digital certificate keys, as well as the storage of those keys, thereby becomes a highly sensitive function that needs to project security both in fact and in appearance in order to gain the confidence of the online merchants and other organizations that commission Symantec's certificate authority services. Last year, a competing certificate authority ran into trouble on the security front and filed bankruptcy a short time later. Once trust is gone, it becomes a difficult thing to restore.

"Through our business model as a PKI service provider, we are asking customers to outsource a critical security function," said senior manager Ralph Claar, who became an expert in cryptography during a previous career in the U.S. Navy. "So we need to instill a high level of trust that we can adequately protect their data and information. So we need a very strong and robust security infrastructure."

There are 13 data centers that check on average of 4.5 billion daily certificate validation queries on a 24/7 basis. Though individual data centers are sometimes taken off-line for maintenance, the overall network has maintained 100 percent uptime since 2004.

A network operations center monitors not only the internal systems, but also the health of the Internet. If there is an outage somewhere in the world, the Symantec NOC is bound to see it, and be able to answer customer queries on ISP outages while generally monitoring the services that are supported by Symantec.

The facility in Mountain View, Calif., is a backup to the primary facility located in Delaware. Symantec spent more than $11 million on security in this building, which also houses the overall network's disaster recovery systems.

Security features include cameras, card keys and pins, firewalls, IDS systems, biometric scanners, human guards, and iron grids in the walls that extend from the concrete slab to the actual roof, thereby making an attempt to burrow through the wall a futile exercise. Server racks were secured with military-grade locks with combinations known to only six people. A diesel generator is constantly on standby to defend against any power outages. Even the redundant systems seemed to have redundant systems.

"We also have solutions in place that aggregate event data from the various devices and systems, and correlate those events to look for anomalous or unauthorized behavior within the environment," said Hans Gustavson, operations director, Trust Services Infrastructure Operations.

NEXT: Redundancy Is King

"The redundancy ties back to our requirement for availability and performance," explained Gustavson. "Because of the critical nature of our service, we strive to maintain 100 percent uptime for our services. So to that degree we have implemented redundancy around mechanical, electrical and building functions, as well as with the network and other compute/storage functions. We have implemented all of these solutions in order to maintain service regardless of whether we undergo planned maintenance or have unplanned issues."

The deeper you travel into the installation, the more strenuous the security precautions become. When you reach the most sensitive areas, such as the data center and the so-called, "key ceremony room," card keys and PINs are supplemented by fingerprint readers and iris scanners. And in order to gain entry, more than one authorized person must be checking into the room before the door will open.

Employees who are authorized to enter the data center are, by default, not authorized to enter the ceremony room unless they are accompanied by an employee specifically authorized for that location. The reverse is also true. By establishing this sort of human firewall, the risk of any internal malfeasance is thereby reduced. To further eliminate the risk associated with potential internal threats, an extended background check is necessary for any employees who would enter those locations, and that status needs to be maintained on an ongoing basis.

"It's as strenuous as we can be without being a governmental agency," said Gustavson. "It's the most rigorous background check that can be done as a commercial entity."

It is called the ceremony room because it is the location where customers are brought for the creation of keys used to support their online certificates.

"That is the room where the key creation takes place and all of those lifecycle events related to the CA creation, the root certificate creation," explained Claar. "So we have to ensure that no one is installed on the malicious code on machines used to generate those keys. It's all about maintaining the integrity."

A lengthy and detailed script is developed for each meeting, and that script must be followed on a line-by-line basis. Meanwhile, a series of cameras records the entire event, including the people in the room, the keys themselves, and shots of the computer screens. "All of this is designed so that we could take it into a courtroom if we needed to," said Claar. "But we've never had to do that in the past."

NEXT: The Safe Room

Extending from within the ceremony room is a separate room housing a series of safes where the actual key devices are stored. The safes are designed to sustain temperatures less than 140 degrees Fahrenheit, even if everything around them should burn to the ground. Similar to the ceremony room, this room, too, can only be entered by a minimum of two authorized people. The system clearly tracks the numbers closely. Although three authorized people were present for our tour, when two of them tried to enter the safe room, access was denied because their departure would have left only one authorized person in the ceremony room itself.

Access to the root key and the intermediate keys require the use of a series of colorful plastic keys that resemble children's toys. These are held by different individuals from the client organization, as well as Symantec. It takes at least three of these keys to gain access to the intermediate and root keys.

An extensive logging process is also in place for virtually every function, including the addition of new devices, the removal of devices, etc. Also, when authorized individuals leave the sensitive areas, they are required to check out of those areas in much the same way that they checked in. Failure to do so would trigger a notification to security guards who would then check the video feed, because the guards, themselves, are not authorized to enter the room either.

Computing and networking hardware is nothing out of the ordinary, given that Symantec prefers to be able to replace physical systems quickly and easily, using off-the-shelf gear. But security and software are often different from what you would find through your typical distributor. Any hardware used in this facility must be approved by NIST.

"It's not what we use, it's how we use it," summarized Claar.

PUBLISHED AUG. 16, 2012