Researchers Discovered More Than 21,000 Container Orchestration and API Management Systems Exposed to the Public Internet
Public cloud and container technology is increasingly used by IT people because of the ease and speed of deployment, ephemeral workloads, and the ability to scale quickly and easily — basically, the agility that public cloud and containers brings to DevOps. Popular container orchestration systems include Kubernetes, Docker Swarm, OpenShift and Mesosphere.
Container clusters are commonly managed, or orchestrated, from administrator dashboards that provide a single interface to manage all aspects of the containers. Kubernetes says it is comprised of a set of independent, composable control processes that continuously drive the current state towards the provided desired state. It says it eliminates the need for orchestration; but also says it orchestrates computing, networking, and storage infrastructure on behalf of user workloads.
The point, however, is that all the different container orchestrations provide a single administrative dashboard for administrator control. This dashboard can spin up new containers, delete unwanted containers, and access both the compute power and stored data on every container. Rarely, if ever, should this dashboard need to be visible to the internet.
“In early June 2018,” states Lacework in a study (PDF) released Tuesday, “Lacework discovered more than 21,000 container orchestration and management systems on the internet, and these results highlight the potential for attack points caused by poorly configured resources, lack of credentials, and the use of non-secure protocols.”
The issue here is that if a system is exposed to the internet when it need not be — indeed should not be — then it is likely that is inadequately secured. If it has been configured to require multi-factor authentication, then access to the orchestration system will have some defense. But, “simple authentication simply won’t be good enough,” Lacework’s chief security architect, Dan Hubbard, told SecurityWeek: “if you have ‘admin’ and ‘abc123’, anyone who can find the admin panel will be able to crack it.”
The report notes that most of the management nodes it discovered are configured to require access credentials, but adds, “These organizations, and the others who will replicate their mistakes, are opening themselves up to brute force password and dictionary attacks.”
It is generally considered that password security on its own is ineffective. But an insecure container orchestration system exposed on the internet is a bigger threat to a company’s cloud-based container infrastructure than a backdoor into a traditional data center. The latter might provide access to a single server under the oversight of the security team and with additional security controls within the data center infrastructure. A container orchestration dashboard, however, provides immediate access to every container in the swarm.
“Consider what would happen,” warns the report, “if [attackers] had all this but could operate their attack all from the Internet, hiding behind proxy servers, VPN concentrators, and compromised routers, essentially masking who they are and where they are coming from. Basically, your data, your customer’s data, and the foundation on which you’ve built your organization would be in major trouble.”
Hubbard believes the basic problem that leaves orchestration dashboards visible to any potential attacker is a disconnect between the DevOps team and the security team. Faced by the need for development speed, DevOps uses containers for all the right reasons. But containers are outside of the traditional security perimeter; and the security team may not even be aware of their use.
“We need to build a bridge from DevOps to security if one doesn’t already exist — that’s the missing piece here,” said Hubbard. “It’s about communicating and making sure that the security team has the tools to know when this is happening. It’s a combination of communication, working together, and for the security team making sure they know when it is happening and/or can detect and handle it when it does happen.”
Exposed container orchestration dashboards are potentially far more dangerous than the more frequently discussed misconfigured S3 buckets. S3 buckets are simple storage devices. Misconfiguration of S3 buckets gives unauthorized access to the stored content, but gives no access to any compute capability.
“The most common attack we are seeing now,” explained Hubbard, “is that attackers are finding these open servers and they are going in there without the company being aware. The attackers are installing their own software and they’re starting new machines and new containers in order to do bitcoin mining. They’re doing this through these open panels they find on the internet. In this scenario the bad guys are getting access to your compute — on all the machines — and they can install their own code and run whatever they want; and they can access your data. So, it’s a lot more powerful than just finding a mis-configured S3 bucket.”
“Let’s be clear,” says the Lacework report. “We are BIG BELIEVERS in all things public cloud, but we need to raise the bar, and raise it quick.” The important thing here is to remove the orchestration panels from the internet.
“You should be able to connect securely through another way,” advises Hubbard, “whether it’s through a central server or through a direct connection through a VPN. Also, depending on the management technology… if its K8 or whatever… they all have different defaults that they ship with and different ways you can configure authentication. Of course, MFA is the best.”
The high-level message from this report is that if you are a developer deploying in the public cloud, then you have a responsibility to think about security; and if you are a security person, and you think that your company is or might be deploying in the public cloud, it’s your responsibility to find out and then to deploy technologies and processes around that to make sure that you are secure. At the moment, there are too many examples of this not happening.
Related: Coverity Scan Hacked, Abused for Cryptocurrency Mining
Related: Container Deployment Grows, Security Concerns Linger: Survey
