Debugging the destroy container on Kubernetes

Keywords: Go github Ubuntu

TL;DR

Content of this article:

  • Describes the, functions and simple use of the destroy image
  • How to debug against the of the destroy container
  • Use of temporary containers (v.1.18 +)

Destroy mirror

The destroy container is used as the name suggests Destroy mirror A container that runs as a base image.

The "destroy" image contains only your application and the dependencies it needs to run. It does not include package managers, shells, or other programs that you can find in the standard Linxu distribution.

GoogleContainerTools/distroless Provides a destroy image for different languages:

What's the use of a destroy image?

Those may be needed when building images, but most are not needed at run time. That's why The last article introduced Buildpacks The stack image of a builder includes the basic image at build time and the basic image at run time, which can minimize the image.

In fact, controlling the volume is not the main function of the destroy image. Limit the contents of the runtime container to the dependencies required by the application, and nothing should be installed. This method may greatly improve the security of the container and is also the most important role of the destroy image.

Instead of delving further into the troubleshooting image, here is how to debug the troubleshooting container

Without the package manager, you can no longer use package management tools like apt and yum after the image is built; Without a shell, you cannot enter the container after the container runs.

"Like a room without any doors, you can't install doors." the destroy image not only improves the security of the container, but also increases the difficulty of debugging.

Using the destroy mirror

Write a very simple golang application:

package main

import (
	"fmt"
	"net/http"
)

func defaultHandler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Hello world!")
}

func main() {
	http.HandleFunc("/", defaultHandler)
	http.ListenAndServe(":8080", nil)
}

For example, gcr.io/destroy/base-debian11 is used as the basic image of golang applications:

FROM golang:1.12 as build-env

WORKDIR /go/src/app
COPY . /go/src/app

RUN go get -d -v ./...

RUN go build -o /go/bin/app

FROM gcr.io/distroless/base-debian11
COPY --from=build-env /go/bin/app /
CMD ["/app"]

Creating a deployment using a mirror

$ kubectl create deploy golang-distroless --image addozhang/golang-distroless-example:latest

$ kubectl get po
NAME                                READY   STATUS    RESTARTS   AGE
golang-distroless-784bb4875-srmmr   1/1     Running   0          3m2s

Attempt to enter container:

$ kubectl exec -it golang-distroless-784bb4875-srmmr -- sh
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "b76e800eafa85d39f909f39fcee4a4ba9fc2f37d5f674aa6620690b8e2939203": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "sh": executable file not found in $PATH: unknown

How to debug the destroy container

1. Use the destroy debug image

Google container tools provides a debug tag for each troubleshooting image, which is suitable for debugging during the development phase. How to use? Replace base image of container:

FROM golang:1.12 as build-env

WORKDIR /go/src/app
COPY . /go/src/app

RUN go get -d -v ./...

RUN go build -o /go/bin/app

FROM gcr.io/distroless/base-debian11:debug # use debug tag here
COPY --from=build-env /go/bin/app /
CMD ["/app"]

Rebuild the image and deploy it. Thanks to the busybox shell provided in the debug image, we can exec to the container.

2. debug container and shared process namespace

Multiple containers can be run in the same pod. Set pod.spec.shareProcessNamespace to true to make the containers in the same pod Multiple containers share the same process namespace.

Share a single process namespace between all of the containers in a pod. When this is set containers will be able to view and signal processes from other containers in the same pod, and the first process in each container will not be assigned PID 1. HostPID and ShareProcessNamespace cannot both be set. Optional: Default to false.

Add a debug container using ubuntu image. Here, for testing (explained later), we add securityContext.runAsUser: 1000 to the original container to simulate the two containers running with different UID s:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: golang-distroless
  name: golang-distroless
spec:
  replicas: 1
  selector:
    matchLabels:
      app: golang-distroless
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: golang-distroless
    spec:
      shareProcessNamespace: true
      containers:
      - image: addozhang/golang-distroless-example:latest
        name: golang-distroless-example
        securityContext:
          runAsUser: 1000
        resources: {}
      - image: ubuntu
        name: debug
        args: ['sleep', '1d']
        securityContext:
          capabilities:
            add:
            - SYS_PTRACE
        resources: {}
status: {}

After updating deployment:

$ kubectl get po
NAME                                 READY   STATUS    RESTARTS   AGE
golang-distroless-85c4896c45-rkjwn   2/2     Running   0          3m12s

$ kubectl get po -o json | jq -r '.items[].spec.containers[].name'
golang-distroless-example
debug

Then enter the pod through the debug container:

$ kubectl exec -it golang-distroless-85c4896c45-rkjwn -c debug -- sh

Then execute in the container:

$ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 14:54 ?        00:00:00 /pause                      # infra container
1000         7     0  0 14:54 ?        00:00:00 /app                         # Original container, UID 1000
root        19     0  0 14:55 ?        00:00:00 sleep 1d                 # debug container
root        25     0  0 14:55 pts/0    00:00:00 sh
root        32    25  0 14:55 pts/0    00:00:00 ps -ef

Trying to access the process space of process 7:

$ cat /proc/7/environ
$ cat: /proc/7/environ: Permission denied

We need to add the following to the debug container:

securityContext:
  capabilities:
    add:
    - SYS_PTRACE

Then access is normal:

$ cat /proc/7/environ
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binHOSTNAME=golang-distroless-58b6c5f455-v9zkvSSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crtKUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443KUBERNETES_PORT_443_TCP_PROTO=tcpKUBERNETES_PORT_443_TCP_PORT=443KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1KUBERNETES_SERVICE_HOST=10.43.0.1KUBERNETES_SERVICE_PORT=443KUBERNETES_SERVICE_PORT_HTTPS=443KUBERNETES_PORT=tcp://10.43.0.1:443HOME=/root

Similarly, we can access the file system of the process:

$ cd /proc/7/root
$ ls
app  bin  boot  dev  etc  home  lib  lib64  proc  root  run  sbin  sys  tmp  usr  var

Without modifying the basic image of the container, use pod.spec.shareProcessNamespace: true to add sys in the security configuration_ The ptrace feature gives the debug container full shell access to debug applications. However, modifying YAML and security configuration is only suitable for the test environment, which is not allowed in the production environment.

We need to use kubectl debug.

3. Kubectl debug

kubectl debug can perform different operations for different resources:

  • Load: create a copy of the running Pod and modify some properties. For example, use a new version of tag in the copy.
  • Load: add a temporary container for the running Pod (described below). Use the tools in the temporary container for debugging without restarting the Pod.
  • Node: create a Pod on the node, run in the node's host namespace, and access the node's file system.

3.1 temporary containers

Since Kubernetes 1.18, you can use kubectl to add a temporary container for running pod s. This command is still in the alpha phase, so you need to "feature gate" Open in.

When creating a k3s cluster using k3d, open the EphemeralContainers feature:

$ k3d cluster create test --k3s-arg "--kube-apiserver-arg=feature-gates=EphemeralContainers=true"@

Then create a temporary container. After creation, you will directly enter the container:

$ kubectl debug golang-distroless-85c4896c45-rkjwn -it --image=ubuntu --image-pull-policy=IfNotPresent
#Temporary container shell
$ apt update && apt install -y curl
$ curl localhost:8080
Hello world!

It is worth noting that the temporary container cannot share the process namespace with the original container:

$ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 02:59 pts/0    00:00:00 bash
root      3042     1  0 03:02 pts/0    00:00:00 ps -ef

You can attach the temporary container to the target container by adding the parameter -- target=[container]. Unlike pod.spec.shareProcessNamespace, the process with process number 1 is the process of the target container, and the process of the latter is the process of the infra container / pause:

$ kubectl debug golang-distroless-85c4896c45-rkjwn -it --image=ubuntu --image-pull-policy=IfNotPresent --target=golang-distroless-example

Note: the current version does not support deleting temporary containers. Please refer to issue , supported versions:

3.2 copy Pod and add container

In addition to adding temporary containers, another way is to create a copy of the Pod and add a container. Note that this is an ordinary container, not a temporary container. Notice that -- share processes is added here

$ kubectl debug golang-distroless-85c4896c45-rkjwn -it --image=ubuntu --image-pull-policy=IfNotPresent --share-processes --copy-to=golang-distroless-debug

Note that -- share processes is added here, and pod.spec.shareProcessNamespace=true will be automatically added:

$ kubectl get po golang-distroless-debug -o jsonpath='{.spec.shareProcessNamespace}'
true

Note: using kubectl debug for debugging does not automatically add sys to pod_ Ptrace security feature, which means that if the UID s used by the container are inconsistent, the process space cannot be accessed. By the time of issuance, Planned to be supported in 1.23.

summary

At present, all of the above are not suitable for use in the production environment, and can not be debugged without modifying the Pod definition.

It is expected that sys will be added to the debug function after kubernetes version 1.23_ Ptrace support. Then try again.

The article is unified in the official account of the cloud.

Posted by asmon on Wed, 03 Nov 2021 00:07:57 -0700