If we run cloud native workload balancing facilities, we can better protect our services. After all, services are often exposed to the public and workloads may belong to different tenants. In this blog post, I will show you how attackers accessing our Kubernetes cluster can make container escape: run Pod to obtain root permission, escape Pod to the host, and continue the attack through invisible Pod and file free execution. At the same time, I will show you how to detect attacks based on Isovalent Cilium Enterprise.
problem
During the container escape, the attacker breaks the isolation boundary between the host and the container, and finally escapes to the Kubernetes control plane or work node. In this case, attackers can see other containers running on the same host, collect their confidential information, read and write data on the host file system, attack Kubelet and upgrade their privileges; Or deploy an invisible Pod to take advantage of Kubernetes errors and persist in the environment.
Applying security best practices in Kubernetes environment can limit these types of attacks, but container breakthrough is still possible. Attackers can use privileged Pod or exploit existing vulnerabilities to obtain privileges. The security team needs to measure whether the enhanced configuration is appropriate and whether the protection of the application is effective.
Solution
One way to achieve this is to follow the observability of the data-driven approach: collect data from Kubernetes workloads and hosts, observe feedback, and make continuous data-driven decisions to protect the Kubernetes environment.
By using eBPF, the security team can directly gain unique visibility into any Kubernetes workload, such as a Pod. Since the Pod on the Kubernetes node shares a kernel, each process in the Pod is visible to a single eBPF program. This provides complete visibility into each process running on the node, whether it is a long-running process on a host managed by systemd or a short-term process running within a container.
Cilium
Cilium uses eBPF to very effectively monitor all network and process behaviors inside Kubernetes workload and outside the host, and provides us with Kubernetes identity awareness and operating system level process visibility of these behaviors.
Cilium exists in the Kubernetes environment in daemon mode. Therefore, there is a cilium agent running on each Kubernetes node, which communicates with the Kubernetes API server to control the Kubernetes Pod identity, network policies, services, etc. Then, based on the identity of each workload deployed in the Kubernetes environment, cilium installed an efficient eBPF program to track connectivity, observability and security for these workloads.
Rich security events
Cilium is able to observe and enforce behavior that occurs within a Linux system. It can collect and filter security observability data directly in the kernel, export it to user space as JSON events, and / or store them in a specific log file through a daemon named bubble enterprise. These JSON events are enriched by Kubernetes identity aware information (including services, tags, namespaces, pods and containers) and operating system level process visibility data (including process binaries, PIDs, UIDs, and parent binaries with a complete process ancestor tree). Then, these events can be exported in various formats and sent to external systems, such as SIEM, such as Elasticsearch, Splunk or stored in S3 bucket. For simplicity, they will be used directly from the log file in this blog post.
Through Cilium's use of real-time network and process level visibility data in the kernel, security teams can view all processes executing in their Kubernetes environment, which helps them make continuous data-driven decisions and improve the security of their systems.
Let's enter the host namespace
In this example, we use a privileged Pod with a host namespace configuration to represent a container escape attack. As we demonstrate here, this is possible in an enhanced Kubernetes environment. Note that there are many ways to perform a breakthrough. For example, an attacker can also exploit the vulnerability to gain privileges and escape from the container sandbox.
The first and simplest step for an attacker to perform container escape is to start a Pod using the privileged Pod specification. Note: Kubernetes allows this by default, and the privilege flag grants the container all available kernel functions. The hostPID and hostNetwork flags place the container in the host PID and network namespace, so it can view and interact with all processes and network resources. A simple example can be found in the following yaml file:
[administrator@JavaLangOutOfMemory ~ ] % cat privileged.yaml
apiVersion: v1
kind: Pod
metadata:
name: privileged-the-pod
spec:
hostPID: true
hostNetwork: true
containers:
- name: privileged-the-pod
image: nginx:latest
ports:
- containerPort: 80
securityContext:
privileged: trueThen, let's execute the privileged Pod specification:
[administrator@JavaLangOutOfMemory ~ ] % kubectl apply -f privileged.yaml pod/privileged-the-pod created [administrator@JavaLangOutOfMemory ~ ] % kubectl get pods NAME READY STATUS RESTARTS AGE privileged-the-pod 1/1 Running 0 11s
Now, the attacker has a privileged Pod and is running, which grants them the same privileges as the root user on the underlying node. Why is it so powerful? Because Pod has begun to have these functions, including CAP_SYS_ADMIN, which is essentially the "new root" in Linux, can also access all devices on the host. This feature, combined with HostPID, allows attackers to access all deployed container namespaces, so they can interact with and take advantage of any other process or file system on the underlying node where the privileged Pod is deployed.
Using Cilium, the security team can obtain the following process exported to user space by executing the following command_ Exec event to detect any privileged container execution:
[administrator@JavaLangOutOfMemory ~ ] % kubectl logs -n kube-system ds/hubble-enterprise -c export-stdout
{
"process_exec":{
"process":{
"exec_id":"bWluaWt1YmU6MTEzNzkyNjAzMjk3MjoxNzk3OA==",
"pid":17978,
"uid":0,
"cwd":"/",
"binary":"/docker-entrypoint.sh",
"arguments":"/docker-entrypoint.sh nginx -g \"daemon off;\"",
"flags":"execve rootcwd clone",
"start_time":"2021-10-13T12:58:31.794Z",
"auid":4294967295,
"pod":{
"namespace":"default",
"name":"privileged-the-pod",
"container":{
"id":"docker://32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"name":"privileged-the-pod",
"image":{
"id":"docker-pullable://nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36",
"name":"nginx:latest"
},
"start_time":"2021-10-13T12:58:31Z"
}
},
"docker":"32865cff8fef4a9274e9fa1d",
"parent_exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"refcnt":1,
"cap":{
"permitted":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"effective":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"inheritable":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
]
}
},
"parent":{
"exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"pid":17958,
"uid":0,
"cwd":"/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158/",
"binary":"/usr/bin/containerd-shim",
"arguments":"-namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf4 eb80d28b94e273d6d1670a6f721a9a1158 -address /var/run/docker/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc -systemd-cgroup",
"flags":"execve clone",
"start_time":"2021-10-13T12:58:31.698Z",
"auid":4294967295,
"parent_exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==",
"refcnt":2
},
"ancestors":[
{
"exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==",
"pid":2680,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/containerd",
"arguments":"--config /var/run/docker/containerd/containerd.toml --log-level info",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.948Z",
"auid":0,
"parent_exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"refcnt":63
},
{
"exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"pid":2672,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/dockerd",
"arguments":"-H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=virtualbox --insecure-registry 10.96.0.0/12",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.908Z",
"auid":0,
"parent_exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"refcnt":65
},
{
"exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"pid":1,
"uid":0,
"cwd":"/",
"binary":"/usr/lib/systemd/systemd",
"arguments":"noembed norestore",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:39:34.078Z",
"auid":0,
"refcnt":101
}
]
},
"node_name":"minikube",
"time":"2021-10-13T12:58:31.794Z"
}Secondly, they can also see the following information:
1. Kubernetes identity aware information, such as namespace: default, pod Name: privileged the pod, container ID, and tag.
2. Visibility information at the operating system level, such as binary files: / docker-entrypoint.sh
pid: 23715,uid: 0 And parameters: nginx -g \"daemon off".
3. Contain / Usr / bin / containerd shim is the complete process root tree of the direct parent process binary.
4. Container enabled features, including CAP_NET_RAW and CAP_SYS_ADMIN.
Second, an attacker can use kubectl exec to gain shell access to the privileged Pod:
[administrator@JavaLangOutOfMemory ~ ] % kubectl exec -it privileged-the-pod -- /bin/bash
The shell that pops up in the container log after startup is of course an event of interest to the security team. They can get the following process exported to user space through Cilium_ Exec event to detect bash execution. You can see the process information between lines 4 and 11, and Kubernetes identity awareness information between lines 12 and 24.
{
"process_exec":{
"process":{
"exec_id":"bWluaWt1YmU6MTI5NDM3OTU0NzQ3ODoxOTU5NA==",
"pid":19594,
"uid":0,
"cwd":"/",
"binary":"/bin/bash",
"flags":"execve rootcwd clone",
"start_time":"2021-10-13T13:01:08.248Z",
"auid":4294967295,
"pod":{
"namespace":"default",
"name":"privileged-the-pod",
"container":{
"id":"docker://32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"name":"privileged-the-pod",
"image":{
"id":"docker-pullable://nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36",
"name":"nginx:latest"
},
"start_time":"2021-10-13T12:58:31Z"
}
},
"docker":"32865cff8fef4a9274e9fa1d",
"parent_exec_id":"bWluaWt1YmU6MTI5NDMzMzczMTY4NToxOTU4NA==",
"refcnt":1,
"cap":{
"permitted":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"effective":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"inheritable":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
]
}
},
"parent":{
"exec_id":"bWluaWt1YmU6MTI5NDMzMzczMTY4NToxOTU4NA==",
"pid":19584,
"uid":0,
"cwd":"/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158/",
"binary":"/usr/bin/runc",
"arguments":"--root /var/run/docker/runtime-runc/moby --log /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb8 d28b94e273d6d1670a6f721a9a1158/log.json --log-format json --systemd-cgroup exec --process /tmp/runc-process133903661 --console-socket /tmp/pty028492678/pty.sock --detach --pid-file /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb8 d28b94e273d6d1670a6f721a9a1158/a5579f11fb7d75d66213f488bf44e9c37b92c196dee5e94647e6f60c59cf6693.pid 32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"flags":"execve clone",
"start_time":"2021-10-13T13:01:08.202Z",
"auid":4294967295,
"parent_exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"refcnt":2
},
"ancestors":[
{
"exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"pid":17958,
"uid":0,
"cwd":"/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158/",
"binary":"/usr/bin/containerd-shim",
"arguments":"-namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf4 eb80d28b94e273d6d1670a6f721a9a1158 -address /var/run/docker/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc -systemd-cgroup",
"flags":"execve clone",
"start_time":"2021-10-13T12:58:31.698Z",
"auid":4294967295,
"parent_exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==",
"refcnt":4
},
{
"exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==",
"pid":2680,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/containerd",
"arguments":"--config /var/run/docker/containerd/containerd.toml --log-level info",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.948Z",
"auid":0,
"parent_exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"refcnt":65
},
{
"exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"pid":2672,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/dockerd",
"arguments":"-H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=virtualbox --insecure-registry 10.96.0.0/12",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.908Z",
"auid":0,
"parent_exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"refcnt":67
},
{
"exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"pid":1,
"uid":0,
"cwd":"/",
"binary":"/usr/lib/systemd/systemd",
"arguments":"noembed norestore",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:39:34.078Z",
"auid":0,
"refcnt":106
}
]
},
"node_name":"minikube",
"time":"2021-10-13T13:01:08.248Z"
}Third, the attacker can use the nsenter command to enter the host namespace and run the bash command as root on the host.
[administrator@JavaLangOutOfMemory ~ ] % nsenter -t 1 -a bash bash-5.0#
The nsenter command executes the command in the specified namespace. First sign - t defines the target namespace that the attacker wants to go to. Each Linux machine runs a process with PID 1, which always runs in the host namespace. Other command line parameters define other namespaces that the attacker wants to enter. In this case, - a describes all namespaces.
Therefore, the attacker is breaking through the container in various possible ways and running the bash command as root on the host.
The security team can select two processes_ Exec event to identify this breakthrough. In the first event, they can observe the nsenter command executed in line 8 and use the appropriate namespace parameters in line 9 - t 1 -a. They can also see the source Pod name, privileged the Pod, and all Kubernetes identity awareness and operating system level visibility information in line 15:
{
"process_exec":{
"process":{
"exec_id":"bWluaWt1YmU6MTYyNzIwMjkzMjkyMToyMzc0Ng==",
"pid":23746,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/nsenter",
"arguments":"-t 1 -a bash",
"flags":"execve rootcwd clone",
"start_time":"2021-10-13T13:06:41.071Z",
"auid":4294967295,
"pod":{
"namespace":"default",
"name":"privileged-the-pod",
"container":{
"id":"docker://32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"name":"privileged-the-pod",
"image":{
"id":"docker-pullable://nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36",
"name":"nginx:latest"
},
"start_time":"2021-10-13T12:58:31Z"
}
},
"docker":"32865cff8fef4a9274e9fa1d",
"parent_exec_id":"bWluaWt1YmU6MTI5NDM3OTU0NzQ3ODoxOTU5NA==",
"refcnt":1,
"cap":{
"permitted":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"effective":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"inheritable":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
]
}
},
"parent":{
"exec_id":"bWluaWt1YmU6MTI5NDM3OTU0NzQ3ODoxOTU5NA==",
"pid":19594,
"uid":0,
"cwd":"/",
"binary":"/bin/bash",
"flags":"execve rootcwd clone",
"start_time":"2021-10-13T13:01:08.248Z",
"auid":4294967295,
"pod":{
"namespace":"default",
"name":"privileged-the-pod",
"container":{
"id":"docker://32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"name":"privileged-the-pod",
"image":{
"id":"docker-pullable://nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36",
"name":"nginx:latest"
},
"start_time":"2021-10-13T12:58:31Z"
}
},
"docker":"32865cff8fef4a9274e9fa1d",
"parent_exec_id":"bWluaWt1YmU6MTI5NDMzMzczMTY4NToxOTU4NA==",
"refcnt":2
},
"ancestors":[
{
"exec_id":"bWluaWt1YmU6MTI5NDMzMzczMTY4NToxOTU4NA==",
"pid":19584,
"uid":0,
"cwd":"/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158/",
"binary":"/usr/bin/runc",
"arguments":"--root /var/run/docker/runtime-runc/moby --log /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb8 d28b94e273d6d1670a6f721a9a1158/log.json --log-format json --systemd-cgroup exec --process /tmp/runc-process133903661 --console-socket /tmp/pty028492678/pty.sock --detach --pid-file /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb8 d28b94e273d6d1670a6f721a9a1158/a5579f11fb7d75d66213f488bf44e9c37b92c196dee5e94647e6f60c59cf6693.pid 32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"flags":"execve clone",
"start_time":"2021-10-13T13:01:08.202Z",
"auid":4294967295,
"parent_exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"refcnt":2
},
{
"exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"pid":17958,
"uid":0,
"cwd":"/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158/",
"binary":"/usr/bin/containerd-shim",
"arguments":"-namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf4 eb80d28b94e273d6d1670a6f721a9a1158 -address /var/run/docker/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc -systemd-cgroup",
"flags":"execve clone",
"start_time":"2021-10-13T12:58:31.698Z",
"auid":4294967295,
"parent_exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==",
"refcnt":4
},
{
"exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==",
"pid":2680,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/containerd",
"arguments":"--config /var/run/docker/containerd/containerd.toml --log-level info",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.948Z",
"auid":0,
"parent_exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"refcnt":86
},
{
"exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"pid":2672,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/dockerd",
"arguments":"-H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=virtualbox --insecure-registry 10.96.0.0/12",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.908Z",
"auid":0,
"parent_exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"refcnt":88
},
{
"exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"pid":1,
"uid":0,
"cwd":"/",
"binary":"/usr/lib/systemd/systemd",
"arguments":"noembed norestore",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:39:34.078Z",
"auid":0,
"refcnt":127
}
]
},
"node_name":"minikube",
"time":"2021-10-13T13:06:41.071Z"
}By getting the second process_ After the exec event, the security team can detect bash execution on the host namespace, which takes nsenter as the parent process binary. The parent process information can be seen between lines 151 and 177, and the source binary name can be seen on line 8, that is / usr/bin/bash:
{
"process_exec":{
"process":{
"exec_id":"bWluaWt1YmU6MTYyNzIwNjE3NjIzMjoyMzc0Nw==",
"pid":23747,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/bash",
"flags":"execve rootcwd clone",
"start_time":"2021-10-13T13:06:41.074Z",
"auid":4294967295,
"pod":{
"namespace":"default",
"name":"privileged-the-pod",
"container":{
"id":"docker://32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"name":"privileged-the-pod",
"image":{
"id":"docker-pullable://nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36",
"name":"nginx:latest"
},
"start_time":"2021-10-13T12:58:31Z"
}
},
"docker":"32865cff8fef4a9274e9fa1d",
"parent_exec_id":"bWluaWt1YmU6MTYyNzIwMjkzMjkyMToyMzc0Ng==",
"refcnt":1,
"cap":{
"permitted":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"effective":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"inheritable":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
]
}
},
"parent":{
"exec_id":"bWluaWt1YmU6MTYyNzIwMjkzMjkyMToyMzc0Ng==",
"pid":23746,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/nsenter",
"arguments":"-t 1 -a bash",
"flags":"execve rootcwd clone",
"start_time":"2021-10-13T13:06:41.071Z",
"auid":4294967295,
"pod":{
"namespace":"default",
"name":"privileged-the-pod",
"container":{
"id":"docker://32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"name":"privileged-the-pod",
"image":{
"id":"docker-pullable://nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36",
"name":"nginx:latest"
},
"start_time":"2021-10-13T12:58:31Z"
}
},
"docker":"32865cff8fef4a9274e9fa1d",
"parent_exec_id":"bWluaWt1YmU6MTI5NDM3OTU0NzQ3ODoxOTU5NA==",
"refcnt":2
},
"ancestors":[
{
"exec_id":"bWluaWt1YmU6MTI5NDM3OTU0NzQ3ODoxOTU5NA==",
"pid":19594,
"uid":0,
"cwd":"/",
"binary":"/bin/bash",
"flags":"execve rootcwd clone",
"start_time":"2021-10-13T13:01:08.248Z",
"auid":4294967295,
"pod":{
"namespace":"default",
"name":"privileged-the-pod",
"container":{
"id":"docker://32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"name":"privileged-the-pod",
"image":{
"id":"docker-pullable://nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36",
"name":"nginx:latest"
},
"start_time":"2021-10-13T12:58:31Z"
}
},
"docker":"32865cff8fef4a9274e9fa1d",
"parent_exec_id":"bWluaWt1YmU6MTI5NDMzMzczMTY4NToxOTU4NA==",
"refcnt":3
},
{
"exec_id":"bWluaWt1YmU6MTI5NDMzMzczMTY4NToxOTU4NA==",
"pid":19584,
"uid":0,
"cwd":"/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158/",
"binary":"/usr/bin/runc",
"arguments":"--root /var/run/docker/runtime-runc/moby --log /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb8 d28b94e273d6d1670a6f721a9a1158/log.json --log-format json --systemd-cgroup exec --process /tmp/runc-process133903661 --console-socket /tmp/pty028492678/pty.sock --detach --pid-file /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb8 d28b94e273d6d1670a6f721a9a1158/a5579f11fb7d75d66213f488bf44e9c37b92c196dee5e94647e6f60c59cf6693.pid 32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"flags":"execve clone",
"start_time":"2021-10-13T13:01:08.202Z",
"auid":4294967295,
"parent_exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"refcnt":3
},
{
"exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"pid":17958,
"uid":0,
"cwd":"/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158/",
"binary":"/usr/bin/containerd-shim",
"arguments":"-namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf4 eb80d28b94e273d6d1670a6f721a9a1158 -address /var/run/docker/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc -systemd-cgroup",
"flags":"execve clone",
"start_time":"2021-10-13T12:58:31.698Z",
"auid":4294967295,
"parent_exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==",
"refcnt":5
},
{
"exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==",
"pid":2680,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/containerd",
"arguments":"--config /var/run/docker/containerd/containerd.toml --log-level info",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.948Z",
"auid":0,
"parent_exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"refcnt":87
},
{
"exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"pid":2672,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/dockerd",
"arguments":"-H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=virtualbox --insecure-registry 10.96.0.0/12",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.908Z",
"auid":0,
"parent_exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"refcnt":89
},
{
"exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"pid":1,
"uid":0,
"cwd":"/",
"binary":"/usr/lib/systemd/systemd",
"arguments":"noembed norestore",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:39:34.078Z",
"auid":0,
"refcnt":128
}
]
},
"node_name":"minikube",
"time":"2021-10-13T13:06:41.074Z"
}Now, the attacker has reached the host namespace on a node in the Kubernetes cluster and is running bash. In this example, we use a privileged container with a hostPID Association. In the real world, this may also be a non privileged container with its own process namespace, and then try to use kernel vulnerabilities to obtain privileges and break through. What can they do? Attackers can see the containers running on the same controller node, collect secrets related to them, read data from the host file system, attack kubelet and upgrade privileges; Or take advantage of Kubernetes' special behavior and keep breaking through by starting an invisible container. Suppose the attacker chooses the last option.