Overview
In our previous research post, “Securing Kubernetes Clusters by Eliminating Risky Permissions”, we covered five examples of risky permissions that can be manipulated by an attacker. One of them was the creation of Pods.
In this blog post, we cover to different RBAC resources that allow the eventual creation of Pods.
An attacker with access to a Kubernetes Pod containing a token that allows the creation of new Pods, could create many crypto-mining containers without you even noticing it. One way to mitigate the risk is to leverage Kubernetes Role-Based access Control (RBAC) to restrict token privileges and deny Pod creation. However, to mitigate the risk, you have to do more than not specify the “Pods” resource in the RBAC policy.
A Little Background on Pods, Workloads Metadata and Controllers
In order to understand the different creation methods for a Pod, we need to understand some of the basics of Kubernetes API.
A Pod is a group of one or more containers that are deployed together on the same host. A Pod is usually used to run applications with several modules inside different containers. Attackers can utilize Pods to run crypto-mining or phishing containers and can also be used to escalate privileges (see example number 2).
Usually, when assigning a user permission to manage Pods (create, delete, modify, etc.), you need to create an RBAC role containing the “Pods” resource. But this is not the only way to assign permissions related to Pods. There are other resources that allow the creation of Pods using trickery.
As stated in Kubernetes API 1.13, workloads are objects that you use to manage and run your containers on the cluster. Workloads use metadata resources, which are the objects used to configure the behavior of other resources within the cluster.
Workloads will eventually run a container, but to run a container, you will need to run a Pod. It is possible to create a Pod as a standalone object. Kubenretes has controllers that can create and manage multiple Pods, handle replication and rollout and provide self-healing capabilities at cluster scope. For example, if we used a deployment (workload object) controller to create a Pod, if the Pod crashes, the deployment controller will start it again and keep restarting it until it runs without crashing.
All the Ways to Create a Container
In the Kubernetes API, we can find the workload object container. This container object is responsible for container creation but can only be created within the context of a Pod object (different workload object).
The container object can only appear in a PodSpec object. A PodSpec object is the “specification of the desired behavior of the Pod,” which means that it defines the properties of the container. The PodSpec YAML below shows the spec field under which the container object is defined.
When checking what objects contain the PodSpec field’s properties, we will find two objects: Pod and PodTemplateSpec. The first is a workload object and creating it will lead to the creation of a container. The latter is a metadata object that describes the Pod that will be created.
As we continue reviewing object available in Kubernetes, we find that there are eight workload objects (see green objects in Figure 2), seven controllers and one Pod object, which can create a container. Each of the controllers are discussed below in more details.
In the chart below (Figure 2), we can also see the spec objects (DeploymentSpec, DaemonSetSpec, etc.). The spec objects describe the desired state for the object (the green objects – the controllers). For example, the DaemonSetSpec object describe the desired state for the DaemonSet object.
The chart (Figure 2) shows that the flow from a controller (green) uses a spec object (blue) to define its state, a pattern which continues all the way to the last spec object (PodSpec) and, eventually, to the creation of container object.
You can get to this conclusion by reviewing all the workloads in the Kubernetes API. But, as you can see from the chart above, there is also a “special” object, called a PodTemplate object, which might also allow Pod creation.
PodTemplate – What are You?
The PodTemplate (orange) is a metadata object contains the specification (PodTemplateSpec) of a Pod. This specification is included in other objects such as Replication Controllers, Jobs and DaemonSets. Controllers use PodTemplate object to make the actual Pod.
Notice that the chart in Figure 2 shows that this object as an outsider because it is not a controller and doesn’t have the fields necessary to create a container object like the controllers’ objects. At the time of writing this, you cannot use PodTemplate to create Pods directly or by referencing its name in an object. You can only use it to create a Pod template object (see Code snippet 1). If, in the future, it becomes possible to reference the PodTemplate object by its name within objects, for example, ReplicaSet it will allow the creation of a Pod indirectly and then we will have to be aware of it.
apiVersion: v1 kind: PodTemplate metadata: name: pod-by-podtemplate namespace: default template: metadata: name: pod-template spec: containers: - name: container image: alpine command: ["/bin/sh"] args: ["-c", "echo \"malicious code\"; sleep 100"]
Figure 3. PodTemplate YAMLEight Ways to Create a Pod
We now list eight different API objects that, if included in a RBAC policy assigned to a user, will allow the user to create Pods.
1. Pod
The most trivial way to create a Pod is by using the “Pod” object. With this method, the Pod will be created as is and, if it crashes, it won’t start again automatically.
kind: Pod metadata: name: pod-by-pod namespace: default spec: containers: - name: container image: alpine command: ["/bin/sh"] args: ["-c", "echo \"malicious code\"; sleep 100"]
Figure 4. Pod YAML
2. ReplicationController
This controller ensures that there are a specific number of Pod replicas running at any time and, if a Pod does crash, the ReplicationController restarts it. This is the original form of replication in Kubernetes and, while it’s being replaced by ReplicaSets, it is still in wide use.
When this controller is created, it will create Pods based on the number of replicas.
apiVersion: v1 kind: ReplicationController metadata: name: pod-by-replicationcontroller spec: replicas: 3 template: metadata: name: pod-template namespace: default labels: name: selector spec: containers: - name: container image: alpine command: ["/bin/sh"] args: ["-c", "echo \"malicious code\"; sleep 100"]
Figure 5. ReplicationController YAML
3. ReplicaSet
This controller orchestrates individual Pod lifecycle and updates. It is a sort of hybrid; in some ways ReplicaSets are more powerful (i.e. more options for the selector field) than ReplicationControllers and, in other ways, they are less powerful.
It has the same behavior as the ReplicationController. When this controller is created, it creates Pods based on the number of replicas.
apiVersion: apps/v1 kind: ReplicaSet metadata: name: pod-by-replicaset spec: replicas: 3 selector: matchLabels: name: selector template: metadata: name: pod-template namespace: default labels: name: selector spec: containers: - name: container image: alpine command: ["/bin/sh"] args: ["-c", "echo \"malicious code\"; sleep 100"]
Figure 6. ReplicaSet YAML
4. Deployment
This controller manages the state of ReplicaSets and the Pods within it. It is intended to replace ReplicationControllers. It provides the same function (through ReplicaSet) and also the ability to rollout changes and roll them back if necessary.
When it is created, it creates a ReplicaSet, which will create a Pod based on the number of replicas.
apiVersion: apps/v1 kind: Deployment metadata: name: pod-by-deployment namespace: default spec: selector: matchLabels: name: selector replicas: 3 template: metadata: labels: name: selector spec: containers: - name: container image: alpine command: ["/bin/sh"] args: ["-c", "echo \"malicious code\"; sleep 100"]
Figure 7. Deployment YAML
5. DaemonSet
This controller ensures that a single Pod exists on each node in the cluster.
When this controller is created, it creates a Pod on each node in the cluster.
apiVersion: apps/v1 kind: DaemonSet metadata: name: pod-by-daemonset namespace: default spec: selector: matchLabels: name: selector template: metadata: labels: name: selector spec: containers: - name: container image: alpine command: ["/bin/sh"] args: ["-c", "echo \"malicious code\"; sleep 100"]
Figure 8. DaemonSet YAML
6. Job
This controller creates one or more Pods and ensures that a specified number of them successfully terminate. If a Pod fails, it will be restarted until the number of completions is reached.
When this controller is created, it creates a Pod.
apiVersion: batch/v1 kind: Job metadata: name: pod-by-job namespace: default spec: template: spec: containers: - name: container image: alpine command: ["/bin/sh"] args: ["-c", "echo \"malicious code\"; sleep 100"] restartPolicy: Never
Figure 9. Job YAML
7. CronJob
This controller creates Jobs on a time-based schedule. It runs a Job written in Cron format periodically on a given schedule.
When this controller is created, it creates a Pod periodically on a given schedule.
apiVersion: batch/v1beta1 kind: CronJob metadata: name: pod-by-cronjob namespace: default spec: schedule: "*/1 * * * *" jobTemplate: spec: template: spec: containers: - name: container image: alpine command: ["/bin/sh"] args: ["-c", "echo \"malicious code\""] restartPolicy: Never
Figure 10. CronJob YAML that creates a Pod every minute
The YAML in Figure 10 above will create new Pod every minute.
8. StatefulSet
This controller manages stateful applications, which are programs that save client data from the activities of one session for use in the next session. It manages the deployment and scaling of a set of Pods, considers each Pod as unique and provides order to Pod deployment. StatefulSet ensures that the next Pod will not launch until the current Pod reaches a running and ready state.
When it is created, it creates a Pod.
apiVersion: apps/v1 kind: StatefulSet metadata: name: pod-by-statefulset spec: replicas: 3 serviceName: "" selector: matchLabels: name: selector template: metadata: labels: name: selector spec: containers: - name: container image: alpine command: ["/bin/sh"] args: ["-c", "echo \"malicious code\"; sleep 100"]
Figure 12. StatefulSet YAML that creates 3 Pods
Conclusion
We looked at eight different ways to create a Pod:
The ability to create a Pod is risky and, when you need to provide permissions without allowing the creation and modification of a Pod, you need to make sure that none of the eight resources in Figure 13 above appear in those permissions.
If the cluster you manage and secure is large and you wish to scan it to detect risky permissions, including the eight different resources above, please take a look at the open source tool we created, KubiScan.
Reference
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/
