Cybersecurity researchers on Tuesday disclosed a new large-scale campaign targeting Kubeflow deployments to run malicious cryptocurrency mining containers.
The campaign involved deploying TensorFlow pods on Kubernetes clusters, with the pods running legitimate TensorFlow images from the official Docker Hub account. However, the container images were configured to execute rogue commands that mine cryptocurrencies. Microsoft said the deployments witnessed an uptick towards the end of May.
Kubeflow is an open-source machine learning platform designed to deploy machine learning workflows on Kubernetes, an orchestration service used for managing and scaling containerized workloads across a cluster of machines.
The deployment, in itself, was achieved by taking advantage of Kubeflow, which exposes its UI functionality via a dashboard that is deployed in the cluster. In the attack observed by Microsoft, the adversaries used the centralized dashboard as an ingress point to create a pipeline to run TensorFlow images that perform cryptocurrency mining tasks.
"The burst of deployments on the various clusters was simultaneous. This indicates that the attackers scanned those clusters in advance and maintained a list of potential targets, which were later attacked on the same time," Microsoft's Senior Security Research Engineer Yossi Weizman said in a report.
The intrusions also echo similar attacks observed by Microsoft Azure Security Center last April that abused Internet-exposed Kubeflow dashboards to deploy a backdoor container for a crypto-mining operation. Weizman told The Hacker News that there's no evidence that the two campaigns were undertaken by the same threat actors.
The ongoing attacks are said to have used two different TensorFlow images — tagged "latest" and "latest-gpu" — to run the malicious code. Utilizing legitimate TensorFlow images is also a clever design to avoid detection in that TensorFlow containers are prevalent in machine learning-based workloads.
Furthermore, Microsoft said the attackers are able to take advantage of the images to run GPU tasks using CUDA, thereby enabling the adversary to "maximize the mining gains from the host."
"As part of the attacking flow, the attackers also deployed [a] reconnaissance container that queries information about the environment such as GPU and CPU information, as preparation for the mining activity," Weizman said. "This also ran from a TensorFlow container."
The development comes days after Palo Alto Networks' Unit 42 threat intelligence team disclosed a brand new form of malware called Siloscope designed to compromise Kubernetes clusters through Windows containers.
Users running Kubeflow are recommended to ensure that the centralized dashboard isn't insecurely exposed to the Internet, and if deemed necessary, require that they be protected behind authentication barriers.
Microsoft has also published a threat matrix for Kubernetes to better understand the attack surface of containerized environments and assist organizations in identifying current gaps in their defenses to secure against threats targeting Kubernetes.
Earlier this April, the company, alongside other members of Center for Threat-Informed Defense, teamed up to release what's called the ATT&CK for Containers matrix that builds upon the Kubernetes threat matrix to detect "risks associated with containers, including misconfigurations that are often the initial vector for attacks, as well as the specific implementation of attack techniques in the wild."
