Cloudflare has been working around the clock in the past few days to address a critical security problem that led to sensitive customer data getting leaked and cached by search engines.
The uninitialized memory leak was discovered by Google Project Zero researcher Tavis Ormandy, who jokingly said he considered the idea of calling it “Cloudbleed” due to similarities to the OpenSSL bug known as HeartBleed.
Ormandy noticed the leakage on February 17, while working on a fuzzing-related project. He immediately notified Cloudflare and the CDN had an initial mitigation in place within an hour. However, the cleanup effort took several days since Google, Yahoo, Bing and other search engines had cached at least 770 URIs across 161 unique domains containing leaked memory.
According to the expert, the leaked data included passwords, cookies, encryption keys, private messages from dating sites, chat messages, IP addresses and even HTTPS requests.
Researcher Nick Sweeting has compiled a list of potentially affected domains, including major services such as Coinbase, DigitalOcean, Medium, 4Chan, Yelp, Uber, Zendesk, OKCupid and Namecheap. Ormandy also named 1Password, but the password manager reassured users that their data was not at risk.
NowSecure has published a blog post detailing how the Cloudbleed bug impacts mobile applications.
In a blog post describing the incident, Cloudflare CTO John Graham-Cumming explained that the company’s edge servers were running past the end of a buffer and returning memory that contained sensitive information.
Cloudflare said memory leakage may have first occurred in September 2016, when the company enabled automatic HTTP rewrites. Then it got worse after a couple of features, server-side excludes and email obfuscation, were migrated to new parsers this year. The content delivery network has determined that the period with the greatest impact was February 13-18, when one in every 3.3 million HTTP requests going through Cloudflare may have resulted in memory leakage.
Graham-Cumming pointed out that customers’ SSL private keys were not leaked, but admitted that a private key used to encrypt connections between the company’s own machines was compromised.
Cloudflare said there was no evidence of any malicious exploits or information being leaked on Pastebin or other such websites. Google Project Zero said it destroyed the data samples collected during its analysis.
Ormandy was ultimately satisfied with how CloudFlare handled the issues and its detailed incident report. However, the expert believes the CDN’s blog “severely downplays the risk to customers.”
In an email to customers, Matthew Prince, Cloudflare Co-founder and CEO, said the company would notify customers if they discovered any data leaked about their domains during the search, and that they would provide full details on what was found.
“To date, we have yet to find any instance of the bug being exploited, but we recommend if you are concerned that you invalidate and reissue any persistent secrets, such as long lived session identifiers, tokens or keys,” Prince wrote. “Due to the nature of the bug, customer SSL keys were not exposed and do not need to be rotated.”
Related: “Ticketbleed” Flaw Exposes F5 Appliances to Remote Attacks