Microsoft Cloud Outage Blamed On Faulty Update

According to a blog post by a Microsoft vice president Arthur de Haan, an attempted firmware update on the afternoon of the 12th was the source of the problem.

"This is an update that had been done successfully previously, but failed in this specific instance in an unexpected way," he wrote. "This failure resulted in a rapid and substantial temperature spike in the datacenter. This spike was significant enough … that it caused our safeguards to come in to place for a large number of servers in this part of the datacenter.

[Related: The 100 Coolest Cloud Computing Vendors of 2013 ]

"These safeguards prevented access to mailboxes housed on these servers and also prevented any other pieces of our infrastructure to automatically failover and allow continued access," he continued. "This area of the datacenter houses parts of the Hotmail.com, Outlook.com, and SkyDrive infrastructure, and so some people trying to access those services were impacted."

The executive went on to explain that the team was immediately alerted of the outage, and begin working to restore service. "Based on the failure scenario, there was a mix of infrastructure software and human intervention that was needed to bring the core infrastructure back online," he said. "Requiring this kind of human intervention is not the norm for our services and added significant time to the restoration."

Service was restored on an incremental basis from Tuesday night through Wednesday morning, with the majority of mailboxes restored before midnight.

This was the second major outage suffered by Microsoft in recent weeks. On February 23 a glitch involving an expired SSL certificate impacted more than 50 services delivered through Microsoft Azure. Impacted customers have been promised a refund in the Azure incident.

PUBLISHED MARCH 14, 2013