Security Experts:

Connect with us

Hi, what are you looking for?

SecurityWeekSecurityWeek

Cybercrime

Microsoft Identifies Attack Targeting Kubeflow Environments

Microsoft’s Azure Security Center (ASC) recently identified an attack campaign that targets Kubeflow, a machine learning toolkit for Kubernetes.

Microsoft’s Azure Security Center (ASC) recently identified an attack campaign that targets Kubeflow, a machine learning toolkit for Kubernetes.

An open-source project released in 2017, Kubeflow is a popular framework for running machine learning (ML) workflows in Kubernetes, at scale. It is aimed at helping with the deployment of open-source systems for ML to diverse infrastructures.

The observed attack, Microsoft reveals, was aimed at mining for cryptocurrency using Kubernetes clusters, which is not surprising, given the fact that some nodes used for ML tasks are often relatively powerful, and in some cases include GPUs.

In April, an image running an XMRIG miner was observed being deployed from a public repository on many different clusters. The same repository, the tech company says, contains other images with minor differences in mining configuration, and those were observed being deployed as well.

Most of the clusters the image was deployed on would run Kubeflow, which suggested that the machine learning framework was the main access vector in the campaign.

This was likely possible because some users exposed the Istio Service to the Internet, for convenient direct access to a user dashboard (otherwise, they would need to use port-forward for access, and have traffic tunneled via the Kubernetes API server).

“By exposing the Service to the Internet, users can access to the dashboard directly. However, this operation enables insecure access to the Kubeflow dashboard, which allows anyone to perform operations in Kubeflow, including deploying new containers in the cluster,” Microsoft explains.

Once access to the dashboard is available, the attacker can deploy a backdoor container in the cluster using various methods, such as the creation of a Jupyter notebook server (and then select the custom image to run on it), or the deployment of a malicious container from an existing Jupyter notebook.

Since Kubeflow is a containerized service, meaning that tasks run as containers in the cluster, an attacker would only need to gain access to Kubeflow to run a malicious image.

“The attacker used an exposed dashboard (Kubeflow dashboard in this case) for gaining initial access to the cluster. The execution and persistence in the cluster were performed by a container that was deployed in the cluster. The attacker managed to move laterally and deploy the container using the mounted service account. Finally, the attacker impacted the cluster by running a cryptocurrency miner,” Microsoft notes.

The tech company also provided details on how one can check whether the malicious container was deployed in a cluster or not. Making sure that Kubeflow’s dashboard isn’t exposed to the Internet should keep deployments safe from this and similar attacks.

Related: Vulnerability in Kubernetes Allows Access to Custom Resources

Related: NSA Shares Guidance on Mitigating Cloud Vulnerabilities

Related: Venafi Acquires Kubernetes Services Provider Jetstack

Written By

Ionut Arghire is an international correspondent for SecurityWeek.

Click to comment

Expert Insights

Related Content

Cybercrime

Zendesk is informing customers about a data breach that started with an SMS phishing campaign targeting the company’s employees.

Cybercrime

The release of OpenAI’s ChatGPT in late 2022 has demonstrated the potential of AI for both good and bad.

Cybercrime

The FBI dismantled the network of the prolific Hive ransomware gang and seized infrastructure in Los Angeles that was used for the operation.

Malware & Threats

Microsoft plans to improve the protection of Office users by blocking XLL add-ins from the internet.

Cybercrime

A new study by McAfee and the Center for Strategic and International Studies (CSIS) named a staggering figure as the true annual cost of...

Cybercrime

Video games developer Riot Games says source code was stolen from its development environment in a ransomware attack

Cybercrime

CISA, NSA, and MS-ISAC issued an alert on the malicious use of RMM software to steal money from bank accounts.

Application Security

PayPal is alerting roughly 35,000 individuals that their accounts have been targeted in a credential stuffing campaign.