In July 2015, operations performance management company PagerDuty advised customers to change their passwords after discovering that its systems had been breached.
Recently, it was discovered that the attacker gained access to PagerDuty servers’ through Linode’s (PagerDuty’s cloud provider) administrative panel. To do so, attackers had to bypass the administrative Two-Factor-Authentication (2FA) controls. as a result of the breach, it’s reported that PagerDuty has moved to another cloud provider.
Last week, Linode had published a blog post, which describes the results of their investigation of the breach. Although PagerDuty is not named explicitly, the similarity in details and in dates leaves no room to doubt on the victim’s identity. The blog was viewed by the security community as an odd combination of disclosure and concealment. On the one hand, it seems to provide many previously unknown facts, but on the other it fails to connect them into a coherent story. In this column I will try to fill the gaps and connect the dots.
Time-based One-time Password (TOTP)
Linode implemented the TOTP algorithm as their second authentication factor. Wikipedia explains TOTP as follows: “In a typical two-factor authentication application, user authentication proceeds as follows: a user enters username and password into a website or other server, generates a one-time password for the server using TOTP running locally on a smartphone or other device, and types that password into the server as well. The server then also runs TOTP to verify the entered one-time password. For this to work, the clocks of the user's device and the server need to be roughly synchronized (the server will typically accept one-time passwords generated from timestamps that differ by ±1 time interval from the client's timestamp). A single secret key, to be used for all subsequent authentication sessions, must have been shared between the server and the user's device over a secure channel ahead of time.”
Figure 1 Google Authenticator TOTP
The security of the TOTP algorithm depends on the secrecy of the shared key. If this key is compromised, either through a server-side or a client-side breach, the TOTP becomes worthless, as the attacker is able to generate one-time password, too.
On the PagerDuty breach, the attackers had compromised the Linode credentials of a PagerDuty’s employee including its 2FA to gain access to Linode management interface. Using that interface attacker gained access to PagerDuty’s Linode hosted servers.
PagerDuty was convinced that the source of the credentials compromise was not the client-side (e.g. some mobile malware), as the specific employee had no possession of its 2FA, as its phone was wiepd months prior to the incident.
Consequently, PagerDuty notified Linode of the breach, and was able to indicate the IP address of the attacking server. Surprisingly enough, the attacker’s server was hosted on Linode too, which enabled Linode to obtain a full image of it.
When Linode examined the server image, they found it hold all the tools and data needed to break Linode’s TOTP algorithm, according to their blog post: “we discovered software capable of generating TOTP codes if provided a TOTP key. We found software implementing the decryption method we use to secure TOTP keys, along with the secret key we use to encrypt them. We also found commands in the bash history that successfully generated a one-time code”
Attacking a cloud provider service, from a machine hosted by the very same provider is usually a really bad idea, as it enables the cloud provider to investigate the attacking machine (which Linode eventually did). If the attacker’s machine was hosted by another provider, most chances that such analysis wouldn’t have been possible. Therefore, we need to assume that the attacker was obligated for some reason to launch the attack from a Linode hosted server for it to work.
A possible such reason could have been an internal Linode vulnerability that can only be triggered from within Linode. Linode’s blog says their security team discovered a vulnerability in Lish’s SSH gateway (Lish is Linode Shell, a proprietary software developed by Linode) that potentially could have been used to obtain the information found on the attacker’s machine image. The exact nature of the vulnerability is not disclosed, but the blog’s first specified improvement done by Linode as a result of the breach, is to limit Lish direct access to credential information: “Several of our applications (such as the Linode Manager and Lish) perform user authentication. Previously, these applications performed this function by having access to the credential information directly within our database.”
Summing it all up, a very plausible technical explanation to Linode/PagerDuty breach, is that the attackers abused Lish direct database access, to get access to other users’ credentials stored on the same database (e.g. via SQL injection). However, to abuse Lish, attackers had to operate from within Linode system, hence attacking from a Linode hosted server.
When Two Becomes One
As in most cases, administrative accounts credentials were the weakest link in Linode’s/PagerDuty breach. Linode Two-Factor-Authentication was only relevant for the client-side, as it required something the client knows (password) and something the client have (mobile app with embedded secret). However, on the server-side these two factors “collapsed” into a single factor as both the password and the TOTP’s secret key were accessible from a single application and probably stored on the same user’s row in the database.
Cloud environments introduce both opportunities and risks. Therefore, cloud providers must make security their top priority, and cloud customers must consider the security provided by the different cloud providers as a key element in making their purchasing decision.