The recently-disclosed Heartbleed vulnerability has forced many system administrators to promptly patch affected servers. That was just the easy part. The hard work lies ahead.
Much of the initial rush to fix the Heartbleed vulnerability focused on Web servers, but there are many other products that utilize OpenSSL, the affected cryptographic library that was found to leak data 64kb at a time in response to attackers sending a specially crafted packet.
Organizations have to identify all the affected systems, generate new public-private key pairs, request and issue new certificates signed by the new keys, and revoke the old certificates. While some organizations have revoked the old certificates for their public-facing Web sites, many have not, and progress is slow remediating other affected systems, experts said.
The first question organizations have to grapple with is, “What’s the size of the problem?” and that is not an easy question to answer, said Kevin Bocek, vice-president of security strategy and threat intelligence at Venafi. “OpenSSL is huge.”
The vulnerability is “catastrophic” for SSL and Internet security, Bruce Schneier, a well-known cryptologist and CTO of Co3 Systems, told SecurityWeek earlier this month. “On the scale of 1 to 10, this is an 11.” While it’s perfectly possible there are even more serious flaws in TLS lurking undiscovered, Heartbleed is quite possibly the worst one to date. Calling Heartbleed a “ginormous issue” would be a conservative assessment, Schneier said.
Various Windows and Android applications are at risk. Networking gear such as cable modems, routers, VPN gateways, and network-attached servers may be vulnerable, as are embedded systems such as heating and ventilation systems and consumer electronics. Add in the Internet of Things, such as smart TVs and thermostats, and it’s clear just understanding the scope of the issue is a challenge.
It’s tricky enough keeping up with regular firmware upgrades for many of these devices under normal circumstances. It’s even more difficult when there is a rush to deal with everything, and many embedded systems were not designed with an upgrade path at all.
For vulnerable devices that can be updated, the to-do list sounds straightforward. Administrators have to patch the flaw, by either updating OpenSSL or installing new firmware. Next, new private-public keys have to be generated. Organizations have to assume the certificates and the private keys have been compromised because proving otherwise is nearly impossible, Bocek said.
The next step is to work closely with the certificate authority to issue new certificates signed with the re-generated private key. Administrators then install the certificate and validate it was deployed correctly. Finally, the old certificate needs to be revoked. Some CAs offer self-service tools or APIs to help facilitate the process. Even so, the whole process is time-consuming and cannot be easily automated, Bocek warned.
Key and certificate management is “not a user-friendly task to begin with,” Bocek said.
It’s not that organizations aren’t taking the problem seriously. The issue lies in the fact that organizations frequently do not have a clear understanding of who is using which keys in what application, or even a complete list of all the certificates and keys deployed. In many cases, certificates are manually managed using labor-intensive and error-prone spreadsheets. Add in multiple administrators—large enterprises can easily have hundreds of them—and it becomes difficult to coordinate their activities. It’s fairly easy to see which systems have been patched, but the manual process makes it harder for CISOs to validate all certificates have been reissued, installed, and verified, especially when dealing with so many keys at once, Bocek said.
Netskope analyzed 4,500 enterprise cloud apps and initially determined 100 were vulnerable to Heartbleed. The number of vulnerable enterprise apps has dropped to 29, but note that Netskope’s analysis looked only for patched servers. A Netskope spokesperson confirmed the cloud apps in this analysis still haven’t reissued or revoked certificates.
According to analysis by researchers at FireEye, roughly 150 million downloads of Android apps contained OpenSSL libraries vulnerable to Heartbleed as of April 17.
Patching is critical because it prevents future damage, but it’s only the first step. The second step, to issue new certificates and revoke old ones, is as essential because it mitigates the risk from any keys that may have been stolen before the servers were patched.
“We will see how many actually do that,” Bocek said.
In an analysis of top 10,000 global Websites, Distil Networks found that 84 percent have patched vulnerable systems, 9 percent re-issued certificates, and 15 percent did not. The status of the certificates for the remaining 76 percent was “inconclusive,” the company said.
Internet research firm Netcraft provided a more definitive figure last week, reporting more than 80,000 certificates have already been revoked this month. However, Netcraft also said it knows of more than 500,000 servers running vulnerable versions of OpenSSL, which means a vast majority of the certificates remain at risk.
Netcraft estimated the cost of replacing compromised certificates with new ones at more than $100 million. The final price tag, however, may wind up being less as many CAs are allowing customers to reissue and revoke certificates for free. However, Netcraft also found that many organizations were just buying new certificates rather than re-issuing new certificates. “Perhaps in the haste of resolving the problem, this seemed the easiest approach, making Heartbleed a bonanza for certificate authorities,” Netcraft’s Paul Mutton wrote last week.
It’s vital that organizations revoke old certificates. Attackers can use the old certificates and the compromised keys to create Websites spoofing banking, e-commerce, and other sites. Browsers use certification revocation lists (CRL) maintained by certificate authorities and other organizations to determine a certificate’s validity.
Netcraft found last week that some companies—such as Yahoo’s mobile log-in page, the U.S. Senate large file transfer system, and GeoTrust’s SSL Toolbox—had deployed new certificates but hadn’t immediately revoked the old ones. Some of those certificates were telling browsers they were “good,” according to the company.
If the old certificate remains valid, the browser cannot to differentiate between fake and real sites. “This can lead to all sorts of problems like increased phishing attacks, more elaborate social engineering tactics and rogue sites, to say the least,” Robert Jeffries, a research analyst with the Solutionary Security Engineering Response Team, wrote in a blog post last week.
“Heartbleed is not a warning shot over the bow,” Bocek said. “It hit the engine room. We need systems that can immediately replace broken trust to get the engine running again.”