Using Kubelet Client to Attack the Kubernetes Cluster

In this blog post, we are going to look at the Kubernetes agent, kubelet (see Figure 1), which is responsible for the creation of the containers inside the nodes and show how it can be exploited remotely to attack the cluster. We will review different misconfigurations of kubelet that have been deployed with default settings as part of a Kubernetes installation and how these misconfigurations could eventually open avenues to the Kubernetes cluster as well as several effective mitigation steps.
Highlights of this post include:

• Covering the basics of kubelet, what it is and its primary goal
• Why default Kubernetes installation leaves unsecured kubelet
• Expose the full kubelet API
• Introduce a new CLI client for kubelet named “kubeletctl”
• Review an attack flow on kubernetes through unsecured kubelet
• Show how to configure kubelet in a secure way to better prevent attacks

Figure 1 – Kubernetes Architecture

Overview

Kubelet is the primary “node agent” that runs on each node and is used to register the nodes in the Kubernetes cluster through the API server. After a successful registration, the primary role of kubelet is to create pods and listen to the API server for instructions. The kubelet receives the instructions from the API server and communicates with the container runtime (or a CRI shim for the runtime, a common example is Docker) over Unix sockets using the gRPC, where kubelet acts as a client and the CRI shim as the server (see Figure 2).

Figure 2 – Kubelet communication with Container Runtime Interface (CRI)

While creating new pods, kubelet continues to monitor the state of the pods and the containers inside it and reports back to the API server on a timely basis. This makes Kubelet a lucrative target for attackers attempting to compromise containers within Kubernetes clusters. At the time of writing this blog we were unable to find any public client for kubelet (kubectl is only a client for the Kubernetes API server but not for the kubelet API), so to better understand its inner workings, we created a client that implements its API.

You might ask yourself “what’s the potential of a client like this that requires credentials to communicate with kubelet?” The truth is that in a default kubernetes installation, kubelet does not require authentication and authorization.

Unauthenticated Kubelet

In a default Kubernetes installation, kubelet runs unsecured — leaving it vulnerable for an attack. The reasons it’s not secured is because anyone can authenticate to kubelet by default since it runs with the anonymous-auth flag set to true. Therefore, requests to the kubelet’s API endpoints are not rejected and treated as anonymous requests. Then the request will be authorized, because by default the authorization mode (authorization-mode flag) is set to AlwaysAllow.

From this point on, we will examine what we can do when there is privileged access to kubelet. The first thing that pops in mind is what API can we call and what are the undocumented calls that might be of interest.

Undocumented Kubelet API

When kubectl is used to retrieve the status of pods in a namespace, behind the scenes the API server collects this information from kubelet. But is there a way for a user to communicate directly with a kubelet? Yes, there is! Kubelet exposes an API that can be used directly without connecting to Kubernetes API server. Unfortunately, the Kubernetes website provides very little documentation about the API and the rest of the undocumented APIs can be seen in the source code (the absence of a documented API is an indication that the API is prone to change). We created a full list of kubelet APIs (based on Kubernetes 1.18; we are keeping the table updated here). In Table 1 you can see a snippet of a few undocumented API calls:

Kubelet API	Examples for use	Description
/pods	GET /pods	List the pods in the kubelet’s worker
/run	POST /run/<podNamespace>/<podID>/<containerName> POST /run/<podNamespace>/<podID>/<uid>/<containerName> Body: <command>	Run command in a container
/exec	GET  /exec/<podNamespace>/<podID>/<containerName>?command=<command> POST /exec/<podNamespace>/<podID>/<containerName>?command=<command> GET  /exec/<podNamespace>/<podID>/<uid>/<containerName>?command=<command> POST /exec/<podNamespace>/<podID>/<uid>/<containerName>?command=<command>	Run command using a stream in a container
/configz	GET /configz	Kubelet’s configuration file settings
/debug	GET /debug/flags/v PUT /debug/flags/v (body: <integer>) GET /debug/pprof/<profile>	Debug information

Table 1 – Kubelet API table

To communicate with the kubelet API using some of the above APIs, we can use curl:

curl -k -X <method> https://<node_ip>:10250/<api_command>/<sub_command>

Some of the API commands are more complex to use with curl, therefore we created a client to simplify the process.

Introducing kubeletctl

We created a new open source tool named “kubeletctl” that implements all the kubelet’s API. It was built with the intention to make it simpler to run commands than using curl, and to allow more advanced requests, which we will cover later. It is available on GitHub at this link: http://github.com/cyberark/kubeletctl

Discovery: It all Starts with Port 10250/TCP

Kubelet exposes its API over the default port 10250/TCP and this is one of the things that we will check when attacking Kubernetes cluster. A privileged access to kubelelt’s port, whether as a result of no authentication or as a result of possessing the required permissions, will allow us to list the pods, access them, and maybe even breakout to the host (if one of the containers is privileged).

To search for nodes with opened kubelet’s port we can use kubeletctl scan command on a specific subnet as appears in Figure 3:

Figure 3 – Scanning for open kubelet ports with kubeletctl

Reconnaissance: List Pods on the Worker

After identifying an accessible kubelet API, we will want to check what information we can extract. The most common and important piece of information is the /pods endpoint which will get us the list of pods:

Figure 4 – Kubelet pods from kubeletctl

Remote Code Execution in Containers

Once we have the details about our pods and containers, we can run commands inside them. Kubelet has 3 API calls for executing commands inside a container:

1. /run   → Run command inside the container
2. /exec → Run command inside the container using a stream
3. /cri   → Run command inside the container after a stream was opened by /exec

Using the /run endpoint with curl is simple:

curl -ks -X POST https://<node_ip>:10250/run/<namespace>/<pod>/<container> -d "cmd=ls /""

But using /exec endpoint with curl is more complex because it requires an initial POST request and a follow-up GET with SPDY capable client (or websocket client which is also supported).

In older versions of Kubernetes (v1.9 for example) you could use this request:

curl -k -H "Connection: Upgrade" \ 
        -H "Upgrade: SPDY/3.1" \ 
        -H "X-Stream-Protocol-Version: v2.channel.k8s.io" \ 
        -H "X-Stream-Protocol-Version: channel.k8s.io" \ 
        -X POST "https://<node_ip>:10250/exec/<podNamespace>/<podID>/<containerName>?command=ls&command=/&input=1&output=1&tty=1"

It would then open a stream and response with 302 requests which will look like this:

<a href="/cri/exec/<cri_value>">Found</a>.

You will need to use other tools like wscat to connect the stream and use /cri/exec/<cri_value>?cmd=<command> to execute the command. In more recent Kubernetes versions, we will not get the cri value anymore because it doesn’t send 302 request.

With kubeletctl you can use /run or /exec without the complexity of handling streams and to make it even easier, we added a scan that checks each container, individually, to see if running a command inside it is possible:

Figure 5 – Kubelet pods and containers vulnerable to remote code execution

Then choose the one you want to execute a command inside it:

Figure 6 – Running command with /exec command

Onelinear Attack

kubelectl is not only a client that implements the kubelet’s API, but it also has more advanced capabilities. It can run a command on all the pods inside the nodes without specifying each pod and container individually:

Figure 7 – Running one command on all existing containers by kubeletctl

Using this feature, it’s possible to extract information from multiple containers with one command. But one of the most common things that attackers in Kubernetes environments will do is to search for tokens, so there is a specific command to view the tokens from all the containers:

Figure 8 – Kubeleltctl collects the tokens from all existing containers

With these abilities, we can collect information from multiple containers fast and save some valuable time.

For more information about the different commands or the usage of the tool, you can always visit the README page on GitHub or just use -h or –help switches when running the tool and you will see the description of each command and how to use it.

Mitigation

A simple way to mitigate the attacks we covered in the previous sections is to use well known deployment tools and managed Kubernetes services like AWS EKS, Azure AKS AKS, kubeadm, etc. These deployments are built with a defense in depth architecture, hence “covering-up” for these unsecured settings.

If this is not the case and you are using the Kubernetes default installation or if you just want to verify that kubelet settings on your nodes are secured, there are two important things you can do to prevent the attacks we highlighted in previous sections:

1. Authentication: Disable anonymous requests to the Kubelet server
2. Authorization: Do not allow all requests and enable explicit authorization

The above configurations can be implemented by modifying the kubelet’s settings. These settings can be configured in one of two ways:

1. Using arguments when running the kubelet executable
2. Using arguments taken from the kubelet configuration file

If both are specified, the executable argument takes precedence.

1. Using kubelet’s executable arguments

kubelet settings can be set through executable arguments, in this case we can edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (can be found by running service kubelet status | grep Drop-In) on each worker node and set the below parameters:

1. Set KUBELET_SYSTEM_PODS_ARGS variable to:

--anonymous-auth=false

2. Set KUBELET_AUTHZ_ARGS variable to:

--authorization-mode=Webhook

After these changes, you will need to restart the kubelet service based on your operating system. For example in Ubuntu:

systemctl daemon-reload 
systemctl restart kubelet.service

2. Using kubelet’s configuration file

To find the kubelet’s configuration file, run ps -ef | grep kubelet | grep config or service kubelet status | grep config and search for –config. This file could be in JSON or YAML format depending on your distribution.

Edit the kubelet’s config file /var/lib/kubelet/config.yaml (default location) with the following changes:

1. Set authentication: anonymous: enabled to false
2. Set authorization: mode to Webhook

Here is an example of how it should look after the changes:

apiVersion: kubelet.config.k8s.io/v1beta1 
authentication: 
  anonymous: 
    enabled: false      -> make sure it is set to false 
    ... 
authorization: 
  # mode: AlwaysAllow   -> make sure this line is not exist 
    mode: Webhook       -> set to Webhook

But I still need access to some Kubelet endpoints!

In the event you still need to access to Kubelet’s endpoints, you can create a role with the rules you need. In Kubernetes documentation, you can view all kubelet API resources and sub-resources:

Figure 9 – Kubelet API resources and sub-resouces

In reviewing Figure 9 above, it is possible to set specific permissions for kubelet. For example, granting read access only to the metrics endpoint:

apiVersion: rbac.authorization.k8s.io/v1beta1 
kind: ClusterRole 
metadata: 
  name: metrics-clusterrole 
  namespace: default 
rules: 
- apiGroups: [""] 
  resources: 
  - nodes/metrics 
  verbs: 
  - get 
  - list

x509 Authentication

Assign this role to a user in the cluster, a one with certificate-based authentication X509 to authenticate with kuebelt but it will work only if the following flags are set:

1. The kubelet had been started with –client-ca-file flag or its configuration file (ex. /var/lib/kubelet/config.yaml) had authentication: x509: clientCAFile: < ca.crt_path>
2. The kubernetes API server had been started with –kubelet-client-certificate and –kubelet-client-key (ex. check the API server YAML cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep kubelet-client)

This means that only users with certificates signed with the certificate authority file (ca.crt) will be able to authenticate to kubelet.

Service Account Token Authentication

For cloud-managed clusters (e.g. EKS – Elastic Kubernetes Service, AKS – Azure Kubernetes Service, etc.) it will be more difficult due to the master node not being accessible to the user and therefore also the ca.crt file. In this case, a service account token can be used with the /proxy endpoint of /nodes like that:

curl -k --header "Authorization: Bearer <service_account_token>" https://<master_ip>:6443/api/v1/nodes/<node_name>/proxy/configz

It will return the same information as was used in kubelet but require secured authentication.

Summary

Kubelet has a cardinal role in Kubernetes – it’s the agent that creates the containers and it has full control over any pod running in the node. Therefore, attackers will set it as one of their targets and will scan for clusters with opened access to kubelet. An access to kubelet allows attackers to gather information about the cluster, access to applications inside the containers, and perform lateral movement which can eventually lead to complete compromise of the cluster.

Kubeletctl can be a good option to communicate with kubelet, in particular for developers who need to run some API against the kubelet, blue teams who want to check if they have vulnerable kubelet in the cluster or for red teams who want automate some of its APIs to run on all the pods.

Although most of the common distributions secure kubelet, IT/DevOps teams should make sure their kubelet configuration is not default and do the necessary changes to make it secured.

References

https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/

https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/

https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/#kubelet-authentication

https://www.deepnetwork.com/blog/2020/01/13/kubelet-api.html

https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/server.go

https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-kubelet-823be5c3d67c

https://www.youtube.com/watch?v=gVqa1cNIZT0

https://static.sched.com/hosted_files/kccnceu18/60/cri-second-boom-of-container-runtimes-KC_CNC_EU_2018.pdf

https://gist.github.com/lizrice/c32740fac51db2a5518f06c3dae4944f

https://github.com/ManicodeSecurity/Defending-DevOps/blob/0045e2eb30f3c3abc0ee34391d6f5356d9ce3766/Attacking-Kubelet/lab.md

https://labs.f-secure.com/blog/attacking-kubernetes-through-kubelet/

https://github.com/ManicodeSecurity/Defending-DevOps