Microsoft Blames Firmware Update For Cloud Failure

Microsoft has identified the cause of the access problems to Hotmail, Outlook.com and portions of SkyDrive earlier this month, and it wasn’t bad weather, power grid troubles or even a wayward administrator.

The trio of cloud services suffered extended downtime – although SkyDrive to a lesser extent – due to a glitch that underscores the importance of proper and uninterrupted cooling in data centre environments, particularly those of cloud services providers. While Microsoft did not reveal the exact nature of technical fault, it did involve a firmware update gone awry.

Temperature Rising

On 12 March, Microsoft confirmed reports of the outage on the Live.com service status page. “We’re having a problem accessing email. You might not be able to see all your email messages. We’re working to restore service right now,” reported Microsoft at 5:35 pm.

As the hours passed, the company supplied frequent, but no less cryptic, updates until the matter was resolved 13 March. Now, a clearer picture has emerged.

In a post on Office Blogs, Microsoft Vice President Arthur de Haan explained: “On the afternoon of the 12th, in one physical region of one of our data centres, we performed our regular process of updating the firmware on a core part of our physical plant. This is an update that had been done successfully previously, but failed in this specific instance in an unexpected way.”

Like in finance, Microsoft’s cloud computing team discovered that past performance does not guarantee future results in IT.

“This failure resulted in a rapid and substantial temperature spike in the data centre. This spike was significant enough before it was mitigated that it caused our safeguards to come in to place for a large number of servers in this part of the data center,” de Haan wrote.

Those safeguards may have spared the physical infrastructure some damage, but it caused headaches for users who rely on Hotmail and Outlook.com for their email or SkyDrive for their cloud storage. Not only were inboxes and some online file stores rendered inaccessible, failover operations never kicked in.

de Hann’s post also provides a clue as to why the outage stretched into the morning hours of 13 March.

“Based on the failure scenario, there was a mix of infrastructure software and human intervention that was needed to bring the core infrastructure back online. Requiring this kind of human intervention is not the norm for our services and added significant time to the restoration,” he wrote.

Microsoft is keen to restore confidence in its cloud ecosystem. “Now that we’re through the resolution, we’re also hard at work on ensuring this doesn’t happen again,” de Hann wrote.

A lot is at stake for the software giant as it makes the transition to a software as a service (SaaS) provider.

On 18 February, David Law, Outlook.com director of product management, announced that in the six months since launch, Outlook.com had attracted 60 million active users. And as with most big product releases, Microsoft CEO Steve Ballmer announced the availability of Office 365 Home Premium, a cloud-enabled version of the Office productivity software suite, 29 January.

Test your Microsoft knowledge with our quiz!

Originally published on eWeek.

Pedro Hernandez

Pedro Hernandez covers Microsoft products and services, such as Office, Windows, Windows Phone, Azure and Skype.

Recent Posts

China President Xi Meets With Top Tech Leaders

High-profile meeting with tech leaders seen as signal China is boosting tech sector after years…

10 mins ago

South Korea To Buy 10,000 GPUs For National AI Hub

South Korea hopes to gain leg up in international AI race with infusion of private…

40 mins ago

BYD, Geely, Great Wall Add DeepSeek AI To EVs

Chinese electric vehicle giants rush to incorporate DeepSeek AI tech to cars after it creates…

1 hour ago

South Korea Suspends DeepSeek From App Stores

South Korean data authority suspends Chinese AI start-up DeepSeek from Apple, Google app stores while…

2 hours ago

Google Puts ‘Profits Over Privacy’ With Tracking Change

Privacy advocates criticise Google over decision to allow companies to track users via digital fingerprints,…

2 hours ago

Meta To Build World’s Longest Subsea Fibre Cable

Meta Platforms announces plan to build Project Waterworth, a 50,000 km cable linking five continents…

3 hours ago