Partners Expect Potential Customer Downtime, Months-Long Resource Drain From Cisco Product Replacement Effort
A product replacement effort centered on some of Cisco's most popular routers, switches and firewalls and sparked by a failing clock signal component Cisco has not named will result in network downtime for some customers as channel partners work to swap out faulty gear before it fails, solution providers said.
Cisco solution providers said they are working with customers on a case-by-case basis at their own expense to map out replacement plans and schedule on-site service calls to remove the impacted equipment and install the replacement gear Cisco is providing to eligible customers.
"We're looking at the potential of having some outages for larger customers who aren't as flexible," said one executive from Cisco channel partner who asked not to be identified, noting that his company has discovered over 300 impacted devices among 25 of its customers so far. "We also have clients that are single-threaded, meaning they've got one router, one switch, and any outage on any of that will affect their business depending on what time of the day it happens … So on a case-by-case basis, it could be extremely impactful, and in other cases it could just be a slight blip on the radar."
Solution providers are currently working to identify and contact customers with Cisco products – including Cisco Integrated Services Routers, Nexus switches, Adaptive Security Appliance firewalls and Meraki cloud-managed switches -- that contain a faulty third-party clock signal component that could cause the system to fail after 18 months in production. Cisco said it expects a noticeable increase in product failures after three years of runtime.
San Jose, Calif.-based Cisco on Feb. 2 disclosed the problem and plans to offer replacement products to eligible customers, including those whose products are under warranty, those with service contracts such as Cisco Smart Net and those with no coverage who purchase service contracts for their impacted equipment.
"In an effort to help customers best address any concerns of network downtime, we are providing these products proactively before they could potentially fail. That gives partners and customers the flexibility to schedule when it's most convenient for them," Cisco said in a statement to CRN when asked to comment about the potential for customer downtime during the replacement effort. "Cisco will work closely with customers and partners as part of our support process as customers schedule maintenance windows and open [Technical Assistance Center] support tickets."
Solution providers are planning to send engineers into the field to swap out the affected products on weekends or after-hours wherever possible in order to minimize the impact to customers of network downtime.
Cisco partners said some of the affected products, such as Cisco's ISRs, are mission-critical pieces of equipment for some customers.
"In most cases, [ISRs] are a customers' egress point in and out of their network, or an ISR and a firewall. When you do this [product replacement], you're down for that time," said one top executive from a solution provider ranked on CRN's Solution Provider 500 list, who did not wish to be named, noting that his company has discovered over 500 impacted devices among 50 of its customers so far. "You can't do that during the day and the configuration of an ISR router is complex, so it's not a simple 'copy and paste' to get that back up and running – that's the issue you're going to run into."
"In most of the cases, customers are saying, 'It's going to have to be done after hours in a maintenance window,'" he said. "At this point in time, we're not charging the customer for this work … We're doing a lot of good will on behalf of Cisco."
Solution providers said the number of impacted customers and devices they uncover could become higher, as they're still trying to identify everyone who is affected. Cisco has declined to disclose the number of impacted customers and devices.
While it is providing replacement equipment, Cisco said it will not cover the onsite services labor for customers who will need to replace affected products, even if those customers purchased a Smart Net contract with onsite services. Some solution providers told CRN they will be providing the installation services for free and absorbing the costs themselves.
"If the customer called us and said, 'We need for you to replace our ISR router.' It's probably an $800 to $1,000 engagement that we may have to do for free or look to eventually recoup that cost [from Cisco] in some way … and that's not revenue by the way, that's just recouping the cost," said the executive on CRN's SP 500 list. "We could have [our engineers] doing something else where we would be making 30 to 35 points on that time."
Partners said the time and effort spent on locating affected customers, scheduling a suitable time to swap out the product and actually sending out engineers to do the onsite work is draining their resources and delaying other projects.
"You've got a certain resource pool to work with when it comes to your support staff or your project staff," said one CTO of a Cisco partner, who did not want to be named, noting that his company has discovered around 350 impacted devices among dozens of its customers so far. "You may have six network engineers, maybe three are working on projects and the other three are on support … When you have situations like this when you've got a vendor who has a massive recall -- because that's what it is even though they're not calling it that publically -- you've got to rejigger your strategy to support these clients."
The CTO said even though many of his customers are covered by support services contracts, his company will take a financial hit and delay other projects for customers.
"Yes, there's going to be some cost effect on us because we are moving resources that we would typically place towards project work, so we may have to push back on some projects," he said. "Forecasting it out, we're probably going to take a hit on some projects, maybe pushing some projects back at least a couple of weeks, which hopefully clients are OK with. It could cost us more if clients are not OK with it."
Although the "clock is ticking" on the affected products, partners are hopeful that they'll be able to replace all the affected products for customers before they fail, thanks to Cisco being upfront proactive in its handling of the problem.
"We haven't had a 9-1-1 situation yet, which is the value of Cisco getting out in front of it," said one solution provider executive from a Cisco Gold partner, who did not wish to be identified. "The majority of our clients are impacted, and we have a closing window of opportunity to address this … We'll be going through this process over the next couple of months, if not longer, but we're working on a replacement program and I'm confident we'll be able to replace everything before it fails."