Security Experts:

Connect with us

Hi, what are you looking for?

SecurityWeekSecurityWeek

Disaster Recovery

Google Issues Post Mortem on Gmail, YouTube Outage

Google has blamed a bug in its global authentication system for last week’s outage that affected Gmail, Calendar, YouTube, Meet and multiple other Google services.

Google has blamed a bug in its global authentication system for last week’s outage that affected Gmail, Calendar, YouTube, Meet and multiple other Google services.

The 47-minute outage last Monday, which severely affected operations at workplaces and schools globally, was caused by a bug in an automated quota management system that powers the Google User ID Service.

In a root cause incident report, Google explained that the Google User ID Service maintains a unique identifier for every account and handles authentication credentials for OAuth tokens and cookies.  This account data is stored in a distributed database, which uses Paxos protocols to coordinate updates. 

For security reasons, this service is programmed to reject requests when it detects outdated data.

Google said one of its automated tools used to manage the quota of various resources allocated for services contained a bug that caused error in authentication results, leading to the service outage.

“As part of an ongoing migration of the User ID Service to a new quota system, a change was made in October to register the User ID Service with the new quota system, but parts of the previous quota system were left in place which incorrectly reported the usage for the User ID Service as 0. An existing grace period on enforcing quota restrictions delayed the impact, which eventually expired, triggering automated quota systems to decrease the quota allowed for the User ID service and triggering this incident,” the company explained.

“Existing safety checks exist to prevent many unintended quota changes, but at the time they did not cover the scenario of zero reported load for a single service,” Google added.

The problem “was immediately clear as the new quotas took effect.”  At the height of the incident, Google could not verify that user requests were authenticated and the company confirmed it was seeing 5xx errors on virtually all authenticated traffic. 

“The majority of authenticated services experienced similar control plane impact: elevated error rates across all Google Cloud Platform and Google Workspace APIs and Consoles, the company said.

Written By

Click to comment

Expert Insights

Related Content

Data Breaches

GoTo said an unidentified threat actor stole encrypted backups and an encryption key for a portion of that data during a 2022 breach.

Mobile & Wireless

Technical details published for an Arm Mali GPU flaw leading to arbitrary kernel code execution and root on Pixel 6.

Mobile & Wireless

Apple rolled out iOS 16.3 and macOS Ventura 13.2 to cover serious security vulnerabilities.

Cloud Security

VMware vRealize Log Insight vulnerability allows an unauthenticated attacker to take full control of a target system.

Mobile & Wireless

Apple’s iOS 12.5.7 update patches CVE-2022-42856, an actively exploited vulnerability, in old iPhones and iPads.

Malware & Threats

Microsoft plans to improve the protection of Office users by blocking XLL add-ins from the internet.

Vulnerabilities

Security researchers have observed an uptick in attacks targeting CVE-2021-35394, an RCE vulnerability in Realtek Jungle SDK.

Mobile & Wireless

Two vulnerabilities in Samsung’s Galaxy Store that could be exploited to install applications or execute JavaScript code by launching a web page.