Salesforce Still 'In Degraded State' After One-Day Outage
A persistent Salesforce.com outage that wiped out four hours of data customers entered into their CRMs on Tuesday had still not been fully remediated by Thursday afternoon.
While CEO Marc Benioff personally apologized to one customer on Twitter, Salesforce wouldn't comment on how widespread the outage was, or what regions or services were affected by the database failure linked to NA14 -- one of 45 Salesforce cloud instances in North America.
Several Salesforce implementation partners contacted by CRN said they had not been fielding complaints from customers, but speculated some might be dealing with the problems internally. One partner told CRN a single customer -- one in Brooklyn, N.Y. -- was impacted.
[Related: The 10 Biggest Cloud Outages of 2015]
But gauging by customer complaints on Twitter and some sources familiar with the industry, the world's largest Software-as-a-Service vendor was grappling with a major outage, one possibly affecting thousands of customers and millions of users.
Salesforce's system status webpage said the performance degradation started at 8:41 a.m. Eastern on Tuesday, followed by a "service disruption" less than an hour later, at 9:31 a.m., and was resolved by 5:30 a.m. Wednesday.
The database error compromised the integrity of files, forcing restoration of NA14 from a backup made four hours earlier. All data written to that instance between 5:53 a.m. and 9:29 a.m. EDT on Tuesday was lost.
The malfunctioning instance became an Internet meme later in the day under the #NA14 hashtag.
The instance "continues to operate in a degraded state," according to Salesforce's status page as of Thursday afternoon. For that reason, Salesforce has suspended some functionality, including weekly exports and sandbox copying.
The status page said the cloud software vendor is performing a root cause analysis, and will share details once available.
One solution provider who works with both Oracle and Salesforce.com and who preferred to remain anonymous told CRN that there will be no long-term impact from the Salesforce.com outage. "Why would someone make a $1 million decision on a four-hour downtime?" the solution provider said.
Every software vendor has had some level of downtime in the past few years, the solution provider said. And while Oracle will try to take advantage of the Salesforce.com outage in its sales efforts, especially since Oracle is in its fourth fiscal quarter, the only impact may be from people sitting on the fence anyway, the solution provider said.
"There will be no big swings," the solution provider said. "It costs a lot to change. If you shift your platforms because of an outage, it's probably just one of 12 or 13 other things that are eating at you. The reason people shift to a new platform is for price, performance, reliability, and other issues. The most important is functionality."
There are a lot of customers who have implemented Salesforce.com poorly, the solution provider said, with poor training or poor processes in place.
"Some customers look at the technology and think it's cool because of the cloud," the solution provider said. "They don't put the tools in place to manage it. But this isn't a Salesforce.com issue. It's a perception that the cloud is an easy way to do something."
Eran Farajun, executive vice president of Asigra, a data backup vendor based in Toronto, told CRN the outage should serve as a reminder to channel partners selling Software-as-a-Service that their responsibilities to customers go beyond simply selling a cloud solution.
"This should remind [the channel] that if you're selling a SaaS application and that data just sits there, you don’t absolve yourself of the responsibility of [protecting] your customers, to ensure the data management processes that the customer was doing for data when it was sitting on-prem are extended to the data sitting now in a SaaS vendor's cloud," Farajun told CRN.
Partners that were bundling backup services could have saved customer data lost in the four-hour window before the rollback. By adding value that way, partners are also doing a better job of monetizing their engagements, he said.
"This is exactly where the channel can add value," Farajun said.
The people responsible for at are probably having the worst Tuesday out of anyone.
/**/ /**/
"move to the cloud" they said, "you'll never be down" they said
/**/ /**/
when you say the same thing in each update for over 6 hours it is no longer an update.
/**/ /**/Joseph F. Kovar contributed to this story.