Google Two-Factor Authentication Flaw Led to Breach at CloudFlare, Exposed Other Google Apps for Business Customers
Late last week, news broke that web security and performance startup CloudFlare was attacked, resulting in a hacker being able to successfully redirect web traffic of one of the company’s largest clients.
While CloudFlare was the victim in this attack, the methods used, along with a flaw in Google’s platform, potentially exposed a large number of Google Apps for Business customers.
The attack targeting CloudFlare kicked off by compromising the personal Gmail account of CloudFlare Co-founder and CEO, Matthew Prince—who happened to be an administrator for the company’s Google Enterprise Apps account for CloudFlare.Com, which had his personal email account listed as a recovery email address.
But according to Prince, all CloudFlare company accounts using the CloudFlare.Com domain had two-factor authentication enabled, so even if the attacker was able to reset the password, the two-factor authentication should have protected the account. It didn’t and that’s where an authentication flaw on the side of Google comes into play.
But first, how did that attackers compromise his personal account? It’s an interesting story that involved a rather elementary, yet clever attack, along with what was assumed to be some social engineering through AT&T’s customer support, and a security flaw in Google’s two-factor authentication and account recovery process.
Prince spoke with SecurityWeek over the weekend about the incident and provided a step-by-step account of how things unfolded, as well as some admissions of mistakes CloudFlare made on its part that fueled the attack’s success.
It’s clear that the attacker had CloudFlare and one of its customers in sight for a while, as the original attempt to compromise Prince’s Google account came via an account recovery request back in early May 2012. But the plot thickened late last week when another attempt proved to be successful. While CloudFlare would not confirm which customer had been targeted, it’s widely understood that the target was 4Chan.org, with a hacker that goes by the name of “UGNazi” taking credit.
A random phone call
At 11:39 AM last Friday, Prince received a call on his mobile phone appearing to originate from a phone number in Chico, California. Busy at the time, Prince ignored the call, letting it go to voicemail. At 11:40 AM—just one minute later, Prince received an email from Google saying that his password has been changed.
Responding to the alert, Prince jumped to investigate the situation.
Immediately, maintaining control his Google account turned into a game of cat and mouse, with Prince and the attacker fighting to gain control over the account in real time. “There was about a 45-minute period of time when we were playing cat and mouse,” Prince said. “After sometime, we were able to suspend my personal Gmail account, re-establish control, and then remove my personal email account from any association with CloudFlare.com.”
From there, the CloudFlare team worked to notify customers, lockdown accounts, and investigate the incident in more detail.
But there is more to the story.
According to Prince, “The password used on my personal Gmail account was 20+ characters long, highly random, and not used by me on any other services so it’s unlikely it was dictionary attacked or guessed.”
So how did the attacker get control of his personal account in the first place?
Late Friday, Prince was tipped off by a text message from a friend, telling him, “Check your voicemail, I think it’s been hacked”. As it turns out, Prince’s voicemail message was modified by the attacker in order to receive and record an automated phone call from Google with a audible code that could be used to reset his account.
“I discovered the change late Friday night when someone called me and then texted me to tell me that my voicemail sounded weird,” he said.
Until then, Prince hadn’t connected the two events, but it had then started to make sense.
“They did it at time when I was unlikely to answer the phone,” Prince said. Once the attacker was able to listen to that voicemail which contained a six-digit reset code, they could quickly take control of his Google account.
“It wasn’t merely that my voicemail was re-recorded, the hacker somehow redirected calls to my phone that went to voicemail to go to another voicemail box that wasn’t associated with my account,” Prince explained. “That meant any messages left there I didn’t have access to.”
On Saturday, Prince called AT&T to investigate how this happened, and inquire about some other text messages and calls he had received.
According to the AT&T representative, somehow the attacker managed to switch the voicemail to point to a number not associated with his account. It’s still unclear how the attacker redirected the voicemail box, but it appears that it could have been a social engineering attack through AT&T’s customer support.
After his voicemail was compromised, the attackers initiated the password reset via an automated phone call, a method Google offers to confirm an account holder’s identity based on a previously provided and connected phone number.
Google two-factor authentication flaw
This is where the Google authentication flaw takes place. As explained earlier, CloudFlare.com accounts had been setup to utilize the two-factor authentication service offered by Google, so even if the attacker was able to reset the account password, the two-factor authentication process should have stopped the attacker. It didn’t, and the attackers were able to easily gain access to the CloudFlare email accounts, an authentication flaw that Google has since acknowledged.
Late Sunday, and into Monday, Google confirmed with SecurityWeek that an authentication flaw did exist related to its two-factor authentication process that was used in the attack.
“We fixed a flaw that, under very specific conditions, existed in the account recovery process for Google Apps for Business customers,” a Google spokesperson told SecurityWeek. “If an administrator account that was configured to send password reset instructions to a registered secondary email address was successfully recovered, 2-step verification would have been disabled in the process. This could have led to abuse if their secondary email account was compromised through some other means.”
“We did some dumb things”
While an authentication flaw, social engineering, and questionable account recovery methods all played a part in the attack, CloudFlare admits, in Prince’s own words, that they “did some dumb things” which enabled the attacker to login and modify some customer records to redirect traffic, leading to the attack’s success.
“One dumb thing that we did early on,” Prince said, “was that in order to make sure that emails we sent to customers were performing correctly and that nobody was abusing our email sending process, some administrators within CloudFlare were BCC’d on transactional emails that were sent to customer accounts.”
As a result of this practice, the attackers were able to access a trove of transactional emails that contained limited customer information, such as email addresses and postal addresses. As part of thoseemails, the company was archiving password reset emails that had been sent to customers, a move the company now regrets.
“If we hadn’t BCC’d the password reset emails, [the attacker] wouldn’t have been able to get into our customer’s account,” Prince admitted.
Once the attacker obtained access to CloudFlare email accounts, he/she able to access a password reset. After likely searching for “4Chan” the attacker was able to quickly do a password reset and gain access to 4Chan’s CloudFlare account. From there, the attacker was able to temporarily redirect traffic from 4Chan.org to the attacker’s handle on Twitter.
While the attacker(s) gained access to CloudFlare’s email, and did access the account of one of its customers the company did emphasize that it’s core system was not breached, and that no credit card data was exposed. In fact, no credit card data is even stored on any of CloudFlare’s systems.
“We have found no evidence of unauthorized access to CloudFlare’s core systems or other customers accounts,” Prince said. “In a review of the contents of the email accounts that were compromised, we discovered some customers’ API keys were present. In order to ensure they could not be used as an attack vector, we reset all customer API keys and disabled the process that would previously email them in certain cases to CloudFlare administrator accounts.”
While Google says it has since addressed the issue, the search giant did emphasize the need for users to remain vigilant.
“As always, we recommend best practices for protecting accounts, such as using 2-step verification and avoiding suspicious requests for account information. Also, we recommend that Google Apps for Business customers with multiple administrators on a domain disable the option to send password reset instructions to secondary email addresses. They can use their other administrators to reset accounts if necessary.”
This incident leaves us with some important considerations, especially for users that have a phone number associated with a Google account. For many, it’s important to realize that your Google account may only be as secure as your four-digital voicemail PIN, so even with these recent kinks, adding two-factor authentication is a good idea for an additional layer of security.
Thanks to Matthew Prince for sharing his story and being transparent on the issue through the entire process.
Related: You Call That Phone Hacking?
