Google Reveals Service Outage Cause

Google has delivered on its promise to explain what caused the 30 to 60 minute services outages on 24 January.

The outages affected users of Gmail, Google+, Calendar, Documents and other applications, and Google has officially pinned the blame on a software bug.

Google Apologies

A detailed technical explanation of what caused the problems, which affected users around the globe, was provided after the incident by Ben Treynor, Google’s vice president of engineering, in a 24 January post on The Official Google Blog.

During the outage, “most Google users who use logged-in services like Gmail, Google+, Calendar and Documents found they were unable to access those services for approximately 25 minutes,” wrote Treynor. “For about 10 percent of users, the problem persisted for as much as 30 minutes longer. Whether the effect was brief or lasted the better part of an hour, please accept our apologies – we strive to make all of Google’s services available and fast for you, all the time, and we missed the mark today.”

The problem that caused the service disruption was tracked down to “an internal system that generates configurations – essentially, information that tells other systems how to behave,” wrote Treynor. A software bug was found by the configuration system, which then generated an incorrect configuration that threw the whole thing into chaos, he wrote.

“The incorrect configuration was sent to live services over the next 15 minutes, caused users’ requests for their data to be ignored, and those services, in turn, generated errors,” he wrote. “Users began seeing these errors on affected services at [2:02 p.m.], and at that time our internal monitoring alerted Google’s Site Reliability Team. Engineers were still debugging 12 minutes later when the same system, having automatically cleared the original error, generated a new correct configuration at [2:14 p.m.] and began sending it; errors subsided rapidly starting at this time. By [2:30 p.m.] the correct configuration was live everywhere, and almost all users’ service was restored.”

Now, that the issue has been identified, Google is working to find ways to prevent such an error from happening again, as well as finding ways to speed up the recovery from similar errors, wrote Treynor.

Repair Work

To accomplish that, Google is correcting the original bug that popped up in the configuration generator to prevent recurrence, and is also auditing all other critical configuration generation systems to ensure they do not contain a similar bug, wrote Treynor. Google engineers are also adding more input validation checks for configurations so that a bad configuration generated in the future will not result in service disruption. The company is also enhancing its targeted monitoring to more quickly detect and diagnose the cause of service failure, he wrote.

The 24 January outage began about 2:12pm Eastern Time (US). Users of Google’s Gmail, Google+ Hangouts and other services experienced shutdowns and slowdowns. Reports about the Google service problems lit up social media Websites from many users who often complained vigorously about the disruptions. Some users who tried to use their Gmail accounts during the outage received Server 500 Error messages, and had to try to resend their messages later.

Google’s Apps Status Dashboard quickly contained limited details about the disruptions, including a list of which services were being affected. An explanation about the cause of the problem was not immediately known at that time. The apps dashboard reported that a long list of Google services were experiencing delays and other issues, which also included Google Talk, Drive, Sheets, Slides, Drawings, Sites and Groups, according to the dashboard.

At 3:23pm ET, an hour or so after the original outage, Google posted an update saying that the Gmail issues were fixed. “The problem with Gmail should be resolved,” the note stated. “We apologise for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.” A short time later, at 3:47pm, Google sent an email update to eWEEK saying: “Most of the issues with Google services have been resolved.”

Google has faced similar service problems in the past. In September 2013, the company apologised to users when Gmail services suffered a slowdown of up to 10 hours due to what it called a dual network failure at the time. The slowdowns during that incident were caused by what Google described as a rare two-pronged failure in the company’s network architecture. The disruption occurred when two separate, redundant network paths stopped working at the same time.

Google’s Gmail, now in its 10th year, was launched 1 April, 2004.

Originally published on eWeek.