Anatomy Of A Shortage

Goodwin told the assembled guests at San Francisco’s Moscone Center in April that Cisco understood just how much the past year’s worth of Cisco supply chain woes had hurt them. Cisco understood, Goodwin said, “the impact it’s having on business.”

Goodwin’s apology came as Cisco partners were grappling at that very moment with a nearly yearlong product shortage that had left them without some of their most critical network building blocks. For partners, this meant frustrated customers, lost sales, and in some cases eyeing alternative product lines from rival vendors.

Following Goodwin to the main stage at the Partner Summit, Randy Pond, Cisco’s executive vice president of operations, processes and systems, described what was akin to a perfect storm of problems, all contributing to Cisco’s supply chain issues. There were component shortages galore from Cisco’s Asian suppliers, due to the effects of a government-induced stimulus in China, which had meant a number of former component factory workers were relocated and were unavailable when demand increased. Further, Pond said, the magnitude of the downturn -- and the rapidness of the recovery -- had caught the industry off guard.

Later in the conference, Cisco Chairman and CEO John Chambers admitted that Cisco had “misread” the situation.

By then, many of the Cisco faithful in the audience were knowingly nodding. It isn’t every day that a vendor as large, aggressive and protective as Cisco offers such a public trio of mea culpas. And at that point, many said they just wanted it all to be over: for lead times on key Cisco products to return to normal, for the endless guesswork to become actionable information.

But as of late July there is no sign, at least from Cisco itself, that solution providers are out of the woods yet. “We think the supply agenda will be tight throughout the entire calendar year and maybe well into next year,” said Chambers in the company’s most recent quarterly earnings call.

The $36 billion longtime networking market leader, sources said, was unprepared to deal with the stresses and strains put on its supply chain by the worldwide economic downturn. From its inability to get contract manufacturers to respond quickly to rapidly changing market conditions to systems that were unable to account for customers placing “the same order multiple times,” Cisco found itself essentially caught flat-footed, sources said.

While Cisco was not alone in its supply chain challenges (see “Battling for Components”), its shortages were more pronounced due to demand for its products.

Simon Minett, executive vice president for global operations and logistics at Westcon Group, one of Cisco’s top specialty distributors, said it wasn’t until at least five months after the problems became apparent that Cisco pressed into action.

“It hit broadly and deeply very quickly,” Minett said. “But these things don’t happen overnight. In the first few months of it happening, recovery was predicted. Month after month, that failed to be achieved. This caught Cisco by surprise, and not until they were some ways into it did they finally begin to take it seriously and respond.”

Cisco, in fact, pointed to just how vulnerable it was as a result of its supply chain shortcomings in its first financial disclosure statement on the crisis last November in a 10-Q filing with the U.S. Securities and Exchange Commission.

Cisco cited “multiple ordering” and “risk of order cancellation,” along with other factors, “may cause difficulty in predicting our sales and, as a result, could impair our ability to manage parts inventory effectively.”

If that wasn’t enough, Cisco acknowledged that efforts to improve manufacturing lead-time performance could have a negative impact on the company. “Increases in our purchase commitments to shorten lead times could also lead to excess and obsolete inventory charges if the demand for our products is less than our expectations,” the company wrote in the 10-Q.

Chambers is not taking the supply chain crisis lightly. He has asked his CFO, his top operations executive and his top manufacturing executive to make whatever commitments are necessary to meet customer delivery dates and ensure that Cisco has the right supply chain metrics for not only this quarter or the next, but more than a year down the road. Chambers calls that supply chain full-court press a “relatively inexpensive insurance policy” to once and for all get it right.

All in all, it has been an unprecedented situation for solution providers and for Cisco, long considered one of the best-run companies in the world.

NEXT: The Extent Of The Shortage

The Extent Of The Shortage

CRN interviewed dozens of Cisco solution providers, distribution executives, analysts and other Cisco observers over the past several months, trying to gain an understanding of how wide and deep Cisco’s supply chain shortages were. Many of Cisco’s public records on the subject -- including the 10-Q filings and comments made by top Cisco executives on quarterly earnings conference calls -- paint a challenging picture.

The crisis began last July and at the peak -- during the end-of-year rush to clear out IT budgets -- product lead times had stretched to two, three and even four times normal delivery dates, according to many solution providers. That meant delays of several months had become part and parcel of doing business with Cisco. And even now, one year later, supply chain issues persist.

Cisco and its channel felt the impact of the shortages up and down the product portfolio, but by far the most commonly affected product families -- named by nearly every source CRNinterviewed -- were its popular ASA security appliances, followed by large swaths of the Catalyst switch line, the newer CP-7900 IP phones, and the Nexus switches integral to Cisco-led data center deployments.

Cisco itself declined to identify affected products or what percentage of products affected by lead-time constraints had returned to normal as of July.

Cisco was hardly alone in its supply chain doldrums. Supply chain shortages during and after the 2009 economic meltdown were rampant, from Hewlett-Packard with printers (see “Battling for Components,”) to component challenges for Intel and plenty of issues for Alcatel-Lucent, Lenovo, Dell and other major vendors, according to solution providers.

But Cisco was in a class by itself in terms of the business impact felt by VARs. First of all, the Cisco products in question function as the central nervous system of the networks that keep businesses up and running. If a company’s network goes down, it stands to lose hundreds of thousands of dollars -- if not millions -- in business and productivity. Many solution providers said they felt painted into a corner because they were reluctant to swap equipment from other vendors into an end-to-end Cisco network.

“Of the VARs and distributors I speak with, Cisco was by far the most egregious offender of product availability over the past year, with lead times double what they normally are,” said Brian Alexander, managing director of equity research for technology hardware/distribution/EMS at research firm Raymond James & Associates. “The reason is that they aggressively curtailed production when the downturn began and demand recovered sooner and faster than they expected.”

It was there that the single biggest problem for solution providers emerged: a lack of communication from Cisco, specifically the type of communication that would have protected hard-won deals by giving solution providers the information necessary to do project planning with their customers. Sources described that lack of communication as jeopardizing their trusted adviser role with customers.

“What they told the channel was minimal,” said a channel source with knowledge of the supply chain, product shortages and how Cisco made its solution provider and distribution partners aware of the challenges. “We’d hear about things in e-mails, or one-off conversations, or a few structured calls; it was various formats. But there was nowhere to go where there was a product listing and you could find out what was constrained.”

Did Cisco’s communication come too late? Earlier information certainly would have helped INX, a prominent Cisco Gold partner in Houston that, in a form 10-Q filed with the SEC in November 2009, detailed a product revenue decrease of 21.4 percent for the quarter, “due to unanticipated product availability issues from our key manufacturer supplier, Cisco Systems Inc.”

In an early May interview, Andy Cadwell, INX’s vice president of sales, told CRN that the key problem was “not being able to set customer expectations correctly.”

“But we report to the public,” Cadwell said at the time. “Cisco’s been pretty quiet in the last year about these issues. You weren’t able to find any public comments except that there have been ‘constraints.’ If they’d talked about it more frankly, it would have helped give us a little bit of air cover and helped us be able to set the correct expectations. Cisco as a company hasn’t really acknowledged it publicly. That’s been the problem.”

Cadwell, reached again in late June, said in an e-mail that lead times had “vastly” improved, especially with Nexus, which was the largest share of INX’s backlog.

But the question is just how much have the lead times improved for partners serving the SMB market? Lead times for one Cisco partner on a $500,000 deal stretched out seven weeks (see “A VAR’s Tale of Discontent,”). This even as Cisco has ramped up its drive to capture SMB market share.

“It’s a shame it happened, because Cisco has finally gotten the messaging right around small-business partners, and they’re ready to go with Cisco, and the product isn’t there,” the supply chain source said. “I think it set them back a bit there. I’m sure it did.”

Joshua Lande, senior account manager at Maureen Data Systems, a Cisco Premier Certified partner in New York, suggested that SonicWall and Juniper were scoring big wins against Cisco based on product availability, and much of it probably came in the SMB space.

“[For those customers], we’ll recommend anything we can,” he said. “It’s very easy for us to say, ‘Just put in SonicWall or Juniper firewalls.’ We’re consultants to those guys [SMBs]; they don’t have anyone on staff. But for the bigger guys with millions invested in Cisco, they can’t really go that route, and then we have to seek help from Cisco.”

Rus Healy, CTO of Annese & Associates, a Herkimer, N.Y.-based solution provider, was among many solution providers who told CRN that information for distributors and VARs was in almost as short a supply as the products themselves.

“As a partner, we deal very much with distribution. We count on distribution to give us an update on delivery issues,” Healy said. “For them, it’s so crucial that they know.“ Healy was among attendees at Cisco Live in Las Vegas at the end of June, and during a networking reception for CCIEs and Cisco executives, asked Chambers for an update on the health of the supply chain.

“He said, ‘Well, I’m glad you asked,’ and he gave the same story,” Healy said, referring to what Goodwin, Pond and Chambers described at the Partner Summit some two months earlier. “It doesn’t really matter. The economy in China is sort of academic to us from the standpoint of trying to meet our customers’ needs, right? They spent five minutes of our collective lives telling us the story, but where does that leave us? Because it doesn’t address the problem.”

Meanwhile, Cisco’s Channel Account Managers (CAMs) were critical in helping solution providers weather the crisis. Lande came to rely on his Cisco CAM to help with loaner products, replacement products, alternative solutions, or at the very least, a flow of information.

“We asked about it a lot, why was it caused, and we didn’t get too many answers,” Lande recalled. “We got that this caught them in a bind, but nothing was straight or 100 percent.”

For a damage control comparison, Lande referenced the major outage suffered by Google in May 2009, which affected roughly 5 percent of all Internet traffic over a four-hour period.

“They sent a follow-up and a big letter out to all of their customers and partners explaining what happened, apologizing and saying they would do everything they could in the future to prevent it from happening,” he said. “Cisco didn’t do that. What we heard from the distributors was the same thing: We don’t know. The only reason I knew about what was going on overseas was because one of the CAMs told me.”

Several sources said Cisco did a poor job communicating the extent of the supply chain crisis because the company and its top executive team did not have comprehensive information on the shortages or how to fix them in a timely manner. The sources suggested that because of Cisco’s size and the complexity of its supply chain, that information wasn’t readily available.

Healy does concede that he is “relieved” that Cisco has made changes to fix and mitigate the supply chain issues.

NEXT: The Gray Market Fallout The Gray Market Fallout

One of the most frustrating aspects of the supply chain crisis for solution providers was that the very products they were desperately seeking to meet customer commitments were showing up on the gray market from unauthorized Cisco suppliers seeking premium prices for the scarce products.

Bob Venero, CEO of Future Tech Enterprise, a Holbrook, N.Y., solution provider and Cisco partner, said plenty of competitors had turned to gray market suppliers like Network Hardware Resale.

“We’re losing deals to gray market providers,” Venero said in a June interview. “The gray market offers better availability.”

No question about it, agreed Mike Sheldon, CEO of Network Hardware Resale, who said business has boomed throughout the past year. “Obviously the constraints are with Cisco’s latest and greatest stuff, because that’s what they sell,” Sheldon said. “We’ve seen tremendous demand for all of those, and in some cases, they’ve been selling for over list price. This is the first time I’ve ever seen that since I came here in 2001.”

The ASAs, the lower-end routers and switches (Catalyst 1900, Catalyst 2960S and Xes), some of the 7604 chassis and plenty of the Nexus switch lines were among the most demanded Cisco lines, Sheldon said, with the ASAs “by far the hottest.” Buyers came to him, Sheldon said, noting that when ASA 5505, 5510 and 5520 appliances were at their worst, “I had 12 in stock.”

“Our business is to have what people need. You have to have a plate glass to break in case of emergency,” Sheldon said. “This time it’s happened with supply chain constraints, and projects and jobs could be at risk. We’ve seen bigger companies that maybe wouldn’t have talked to us five years ago buying from us now.” Sheldon said there has been a shift away from Cisco dominance in the market.

“That bias? That no one ever gets fired for buying Cisco? That’s very slowly ebbing,” Sheldon said. “At the end of the day, at least for now, a lot of these customers have a Cisco network, and if you’re human, you’re adding more Cisco stuff. HP is winning often when the customer’s doing a big rip and upgrade, but absent of that, it’s still a real hard battle with Cisco.”

NEXT: Competitors Benefit

Competitors Benefit

Solution providers said the biggest beneficiaries from the Cisco supply chain crisis weren’t only gray market sellers, but also competitors like SonicWall, Juniper Networks and HP.

Among Cisco competitors that deliberately marketed against its supply chain woes, VARs and distributors frequently cited SonicWall as the biggest noisemaker. That’s something SonicWall’s Marvin Blough, vice president of worldwide sales, doesn’t deny. “We saw an opportunity, and went after it,” he said.

“Here’s the thing we found: If your channel doesn’t hear from you what’s going on, you run into a lot of trouble because they do have alternatives and they look at alternatives. That’s true for any vendor,” Blough said. “Whenever the market-share leader creates a void like that, every other vendor is looking at how they can take advantage of it.”

According to Blough, SonicWall saw its inbound calls from customers and channel partners spike as Cisco’s product shortages continued. Partners who carried both Cisco and SonicWall were the first to inquire, Blough said, but plenty of VARs who hadn’t previously carried SonicWall were burning up the phone lines, too. Blough declined to give specific numbers, but said, “The majority of partners that we’ve recruited in the past few years have been Cisco VARs.”

SonicWall didn’t create a specific program to lure Cisco VARs so much as its account reps met with Cisco partners and tailored individual plans to fast-track their training on SonicWall gear.

“We knew that the biggest thing we had to do was get them smart and educated on the SonicWall product line as quickly as possible,” Blough said. “We did not come out and offer free training to every Cisco reseller; that would have been almost a slap in the face to our partners already committed to SonicWall. But it ended up being a situation where you had a Cisco partner you know who’s committed to starting a relationship with SonicWall, and you sat down and tailored something specific to each partner.”

SonicWall was not alone in courting Cisco partners. Distribution executives confirmed to CRN that HP and Juniper weren’t exactly idle.

Juniper isn’t known as a saber-rattler. (“We are aggressively promoting our product and seeing strong growth as a result,” said Blaine Raddon, vice president of channel sales and general business at Juniper, in a statement e-mailed to CRN.) But, as one solution provider put it, it isn’t necessarily warm and fuzzy, either, and has made sure some of its recent, channel-friendly promotions -- such as “Switch to Juniper,” the 60 percent discounting switch trade-up program it launched in mid-April -- are in play.

“No, Juniper doesn’t make a whole lot of waves. That’s not their style,” said the solution provider, a national VAR that carries Cisco and Juniper. “But if you were at the Juniper [J-Partner conference in Phoenix in May], you know. They made sure, and they’ve continued to make sure, partners know that they aren’t struggling with this stuff at all and the timing’s right for another good look at Juniper.”

Even if partners didn’t have to turn to alternate suppliers to fix deals held up by Cisco shortages, they kept that option as a back-pocket trump card.

“Particularly in switches and firewalls, we’re a partner with Juniper because we hold a couple of contracts that require us to do break-fix maintenance,” said Annese’s Healy. “It would have been easy for us to do it.”

At the very least, most solution providers told CRN they had a plan: loaners from Cisco, spot fixes until they could get their hands on the right equipment at the right place, or slotting in products from other vendors in hopes of remedying a problem.

“This did open opportunities for some of the ProCurve stuff, because what HP is going through is rebranding, and they’re gaining a lot of credibility in the space,” said the vice president of a Cisco Silver partner, who asked not to be identified. “HP has a pretty good product set now, and you know as well as I do that salespeople will take the path of least resistance.”

Mark Fabbi, vice president, distinguished analyst and leader of enterprise network infrastructure research at Gartner, puts it another way: The difference in 2010 is that Cisco is no longer the only game in town.

“The attitude out in the market used to be, “I’m frustrated, but I’m beholden to Cisco.” Now there’s frustration followed by ‘I’m going to find something else to do,’ ” Fabbi said. “The difference this time is that Cisco sits in a much more competitive environment, and there is an increasing number of customers that say, ‘I just can’t run my business this way,’ and have turned away from Cisco.”

Solution providers, too, are broadening their horizons, especially with competitors like HP, Juniper and Brocade having emerged as viable data center and networking alternatives.

“We’re seeing loyal Cisco channels broaden out,” Fabbi said. “There are certainly players that have been predominantly Cisco reseller shops starting to expand their kit bag. I see it more at the end of things, with customers, but I know that resellers are also starting to respond.”

NEXT: The Cisco Response

The Cisco Response

Cisco itself declined repeated requests by CRN to interview Goodwin, Pond or another channel or operations executive for this story. In a statement e-mailed to CRN in mid-July, Goodwin offered the following:

“We realize that dealing with the lead time issue these past several months has not been easy for our partners. As mentioned during our last quarterly conference call, as a result of several actions we have taken, we have seen significant lead-time improvement with many affected product families,” Goodwin said.

“Together with our partners we are addressing four major market transitions -- collaboration, video, virtualization and cloud,” he added. “Each of these transitions by themselves is a huge transformation, but together these market transitions create an opportunity unlike any other I have seen in my more than 30 years in the IT industry. We realize that product availability and lead-time predictability are critical components to capturing this massive opportunity and thus are committed to continuing to improve upon our lead times so that together we can redefine the Internet.”

A Cisco spokesman reiterated that lead times were impacted by two factors. One was increased demand and the other was supplier constraint caused by a range of factors, including labor and semiconductor capacity issues as well as industry consolidation.

“In any rapidly shifting supply/demand environment such as the one we have experienced this past year, shifts in lead times, inventory levels and manufacturing outputs will occur,” the spokesman wrote to CRN.

“We remain confident in our suppliers’ ability to replenish and rebuild their inventory levels to meet customer demand moving forward. We continue to work closely with our suppliers to strive to ensure our value chain operates effectively and we are able to meet customer demands and requirements. We have made great progress, and expect to continue to make progress in the months ahead.”

One sign that things are getting better is Cisco’s impressive switching revenue for its third fiscal quarter ended May 1. Cisco, in fact, said switching sales were up a whopping 40 percent from the year-ago period to $3.7 billion. Not only that, Cisco’s earnings for the quarter were up 63 percent to $2.19 billion compared with $1.35 billion in the year-ago quarter. Sales for the quarter were up 27 percent to $10.4 billion compared with $8.2 billion in the year-ago quarter.

Chambers called the quarter “one of the best we have seen in the history of our company.”

NEXT: The Postmortem The Postmortem

If Cisco learns anything from the crisis, solution providers said it should be that effective communication is critical in a shortage so solution providers and customers can do project planning.

“They could have been clearer, sooner to the channel on the delays so that we could clearly plan,” said Robert Betzel, president of Infinity Network Solutions, a Macon, Ga., solution provider. “If they could learn anything to make it better next time, it’s that they can have the confidence to say we’re going to make partners know as soon as possible and let the partners carry it forward. People forgive when they know details. Most of our clients understood.”

Raoul Tecala, who until May was group vice president of the Cisco Alliance and Dimension Data, Cisco’s largest global reseller, said Cisco did as good as could be expected in a difficult situation.

“The quoting and ordering tools from Cisco give you the lead times, and that’s how we found out, so the question becomes, should they have come right out and said, ‘We’re experiencing these issues. You should expect these lead times to lengthen?’ Yes, I suppose that’s true. Could they have forewarned a bit more on lead times? I suppose that’s fair. But do I blame them for it? I don’t think so,” said Tecala.

Tom Adams, managing director of IT Services at Axispoint, a New York-based Cisco Silver partner, agreed with many solution providers that the frustration felt by VARs wasn’t so much the whys of the shortage as to when products would be widely available.

“The one thing that would have been nice to understand is how long it was going on, but I don’t think even they knew what the demand curve was going to be, and didn’t until April and May when things got back into control. Their communication to us was everything they knew,” said Adams.

Whether Cisco has made supply chain improvements that could allow the company to weather another economic downturn without severe shortages remains to be seen.

According to several solution providers and distributors, Cisco has put several executives on the ground in China to have better day-to-day visibility over suppliers.

Responding to a set of detailed questions sent via e-mail, the Cisco’s spokesman said that Cisco is “working with suppliers, including negotiating incentives, to bring on additional capacity in both materials and personnel for high-demand components, making strategic inventory purchases and deploying Cisco personnel to key supplier factories and contract manufacturers to work closely with our partners at monitoring quality and output levels.”

“In addition, we’ve contracted dedicated logistics and transportation arrangements in order to ensure the fastest possible delivery for both inbound components and outbound finished products,” the spokesman added.

Other improvements have been happening over the past year, the spokesman said. According to Cisco, it provided regular updates to its partner lead time tool, and addressed the issues with its advisory boards and councils, including the Cisco Partner Executive Exchange and Partner Operations Advisory Board (POAB).

“In fact, directly following our Q1 FY10 POAB, Randy Pond led the formation of a POAB Partner Communications Sub-Committee to help get direct feedback from these partners on what they needed from Cisco in the form of communications,” the spokesman said.

Quarterly earnings calls, the spokesman said, provided a “broad-scale update.” Cisco is also continuing to direct partners to its online lead times tool on the Cisco Web site, where registered partners can log in and see expected product availability for individual Cisco products.

Cisco further said that according to its field staff, it learned that payouts in its Value Incentive Program (VIP) could be impacted by lead times. As such, Cisco introduced in March a third VIP payment scheduled for payout in August.

On the communication issue, Cisco offers the following:

“Our channel partners are critical to our success. As such, we strive to be as transparent as we can, especially on topics that impact their business. That said, as a publicly traded company there are guidelines as to when and how we can provide financial guidance. Providing updates on our lead times, positive or negative, would have been considered new financial guidance, and as such we were limited as to the types of external communications we could provide.”

Gartner’s Fabbi isn’t so sure that lets Cisco off the hook. “Cisco is still in control of what they disclose. Publicly traded companies make disclosures, good and bad, all the time that are materially affecting,” he said. “They clearly didn’t want to admit this publicly and they’re big enough that they can hide the impacts in their financials. Cisco likes to hide behind these types of regulations as an excuse not to talk or disclose information about their business.”

Ken Dulaney, vice president and distinguished analyst with Gartner and the researcher’s principal Cisco analyst, agreed that Cisco needed to be more forthcoming.

“What did we learn from Toyota and what are we seeing with BP? You have to tell your folks and you have to be honest,” Dulaney said. “If you had supply chain issues and you didn’t communicate them, shame on you.”

Even as they are still grappling with the Cisco shortage, VARs and distributors have done what they always do: adapt.

Holly Garcia, senior director of vendor management at Ingram Micro, said Ingram had invested to carry more Cisco inventory than usual in accordance with the information coming from Cisco. It is also working with Cisco on more frequent forecasting.

“The kudos I would give to them is they were very sensitive to our quarter-end and year-end needs,” Garcia said. “They’ve been open and have engaged with us quite a bit in that regard.”

Westcon’s Minett said regular meetings between his team and Cisco specifically on the supply chain began a few months ago.

“This was an exceptional situation: the length, the depth of the issue, and you’d probably find most situations self-correcting,” Minett said. “They had to take action which they never had to take before, and realize that the standard response to the situation was no longer achieving the normal remedy. In evolutionary terms, you have to have a setback to push advancement, and this is an advancement that’s pushed collaborative planning with Cisco into a new league. Cisco is a more open book now, and a lot more inclusive.”

And a lot more partner-centric, Minett believes.

“They’re looking at distribution as a partner in the supply chain rather than just a customer,” he said. “We’re going to have a far better integrated supply chain discussion going forward than we ever had before. This has made them stronger.”

For some partners, however, mending fences may take a little longer.

“What you’re seeing from them signifies the core internal problem. They haven’t put solving this as a very high priority,” said a Cisco Gold partner. “Who got fired over this? That’s the question I want to ask. Somebody should have gotten fired. If no one is accountable, nothing changes. I don’t think they’re going to do better. Too often the storm blows over and everyone goes back to what they did before. Who’s accountable? No one’s got any skin in the game?”

“You want to clear away all the B.S. and talk brass tacks? Here’s the bottom line: They took a situation that was largely not their fault and made it a hell of a lot worse by not being forthcoming,” said another national Cisco Gold partner, who spoke with CRN in mid-July. “People needed answers. People needed something. For one of the most important technology companies in the world, with one of the broadest and most important channel programs, that’s simply unacceptable. Do better.”

STEVEN BURKE, SCOTT CAMPBELL & JOSEPH F. KOVAR contributed to this story.