Microsoft’s Azure Security Center (ASC) recently identified an attack campaign that targets Kubeflow, a machine learning toolkit for Kubernetes.
An open-source project released in 2017, Kubeflow is a popular framework for running machine learning (ML) workflows in Kubernetes, at scale. It is aimed at helping with the deployment of open-source systems for ML to diverse infrastructures.
The observed attack, Microsoft reveals, was aimed at mining for cryptocurrency using Kubernetes clusters, which is not surprising, given the fact that some nodes used for ML tasks are often relatively powerful, and in some cases include GPUs.
In April, an image running an XMRIG miner was observed being deployed from a public repository on many different clusters. The same repository, the tech company says, contains other images with minor differences in mining configuration, and those were observed being deployed as well.
Most of the clusters the image was deployed on would run Kubeflow, which suggested that the machine learning framework was the main access vector in the campaign.
This was likely possible because some users exposed the Istio Service to the Internet, for convenient direct access to a user dashboard (otherwise, they would need to use port-forward for access, and have traffic tunneled via the Kubernetes API server).
“By exposing the Service to the Internet, users can access to the dashboard directly. However, this operation enables insecure access to the Kubeflow dashboard, which allows anyone to perform operations in Kubeflow, including deploying new containers in the cluster,” Microsoft explains.
Once access to the dashboard is available, the attacker can deploy a backdoor container in the cluster using various methods, such as the creation of a Jupyter notebook server (and then select the custom image to run on it), or the deployment of a malicious container from an existing Jupyter notebook.
Since Kubeflow is a containerized service, meaning that tasks run as containers in the cluster, an attacker would only need to gain access to Kubeflow to run a malicious image.
“The attacker used an exposed dashboard (Kubeflow dashboard in this case) for gaining initial access to the cluster. The execution and persistence in the cluster were performed by a container that was deployed in the cluster. The attacker managed to move laterally and deploy the container using the mounted service account. Finally, the attacker impacted the cluster by running a cryptocurrency miner,” Microsoft notes.
The tech company also provided details on how one can check whether the malicious container was deployed in a cluster or not. Making sure that Kubeflow’s dashboard isn’t exposed to the Internet should keep deployments safe from this and similar attacks.
Related: Vulnerability in Kubernetes Allows Access to Custom Resources
Related: NSA Shares Guidance on Mitigating Cloud Vulnerabilities
Related: Venafi Acquires Kubernetes Services Provider Jetstack