TL;DR
Content of this article:
- Describes the, functions and simple use of the destroy image
- How to debug against the of the destroy container
- Use of temporary containers (v.1.18 +)
Destroy mirror
The destroy container is used as the name suggests Destroy mirror A container that runs as a base image.
The "destroy" image contains only your application and the dependencies it needs to run. It does not include package managers, shells, or other programs that you can find in the standard Linxu distribution.
GoogleContainerTools/distroless Provides a destroy image for different languages:
- gcr.io/distroless/static-debian11
- gcr.io/distroless/base-debian11
- gcr.io/distroless/java-debian11
- gcr.io/distroless/cc-debian11
- gcr.io/distroless/nodejs-debian11
- gcr.io/distroless/python3-debian11
What's the use of a destroy image?
Those may be needed when building images, but most are not needed at run time. That's why The last article introduced Buildpacks The stack image of a builder includes the basic image at build time and the basic image at run time, which can minimize the image.
In fact, controlling the volume is not the main function of the destroy image. Limit the contents of the runtime container to the dependencies required by the application, and nothing should be installed. This method may greatly improve the security of the container and is also the most important role of the destroy image.
Instead of delving further into the troubleshooting image, here is how to debug the troubleshooting container
Without the package manager, you can no longer use package management tools like apt and yum after the image is built; Without a shell, you cannot enter the container after the container runs.
"Like a room without any doors, you can't install doors." the destroy image not only improves the security of the container, but also increases the difficulty of debugging.
Using the destroy mirror
Write a very simple golang application:
package main
import (
"fmt"
"net/http"
)
func defaultHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello world!")
}
func main() {
http.HandleFunc("/", defaultHandler)
http.ListenAndServe(":8080", nil)
}
For example, gcr.io/destroy/base-debian11 is used as the basic image of golang applications:
FROM golang:1.12 as build-env WORKDIR /go/src/app COPY . /go/src/app RUN go get -d -v ./... RUN go build -o /go/bin/app FROM gcr.io/distroless/base-debian11 COPY --from=build-env /go/bin/app / CMD ["/app"]
Creating a deployment using a mirror
$ kubectl create deploy golang-distroless --image addozhang/golang-distroless-example:latest $ kubectl get po NAME READY STATUS RESTARTS AGE golang-distroless-784bb4875-srmmr 1/1 Running 0 3m2s
Attempt to enter container:
$ kubectl exec -it golang-distroless-784bb4875-srmmr -- sh error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "b76e800eafa85d39f909f39fcee4a4ba9fc2f37d5f674aa6620690b8e2939203": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "sh": executable file not found in $PATH: unknown
How to debug the destroy container
1. Use the destroy debug image
Google container tools provides a debug tag for each troubleshooting image, which is suitable for debugging during the development phase. How to use? Replace base image of container:
FROM golang:1.12 as build-env WORKDIR /go/src/app COPY . /go/src/app RUN go get -d -v ./... RUN go build -o /go/bin/app FROM gcr.io/distroless/base-debian11:debug # use debug tag here COPY --from=build-env /go/bin/app / CMD ["/app"]
Rebuild the image and deploy it. Thanks to the busybox shell provided in the debug image, we can exec to the container.
2. debug container and shared process namespace
Multiple containers can be run in the same pod. Set pod.spec.shareProcessNamespace to true to make the containers in the same pod Multiple containers share the same process namespace.
Share a single process namespace between all of the containers in a pod. When this is set containers will be able to view and signal processes from other containers in the same pod, and the first process in each container will not be assigned PID 1. HostPID and ShareProcessNamespace cannot both be set. Optional: Default to false.
Add a debug container using ubuntu image. Here, for testing (explained later), we add securityContext.runAsUser: 1000 to the original container to simulate the two containers running with different UID s:
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: golang-distroless
name: golang-distroless
spec:
replicas: 1
selector:
matchLabels:
app: golang-distroless
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: golang-distroless
spec:
shareProcessNamespace: true
containers:
- image: addozhang/golang-distroless-example:latest
name: golang-distroless-example
securityContext:
runAsUser: 1000
resources: {}
- image: ubuntu
name: debug
args: ['sleep', '1d']
securityContext:
capabilities:
add:
- SYS_PTRACE
resources: {}
status: {}
After updating deployment:
$ kubectl get po NAME READY STATUS RESTARTS AGE golang-distroless-85c4896c45-rkjwn 2/2 Running 0 3m12s $ kubectl get po -o json | jq -r '.items[].spec.containers[].name' golang-distroless-example debug
Then enter the pod through the debug container:
$ kubectl exec -it golang-distroless-85c4896c45-rkjwn -c debug -- sh
Then execute in the container:
$ ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 14:54 ? 00:00:00 /pause # infra container 1000 7 0 0 14:54 ? 00:00:00 /app # Original container, UID 1000 root 19 0 0 14:55 ? 00:00:00 sleep 1d # debug container root 25 0 0 14:55 pts/0 00:00:00 sh root 32 25 0 14:55 pts/0 00:00:00 ps -ef
Trying to access the process space of process 7:
$ cat /proc/7/environ $ cat: /proc/7/environ: Permission denied
We need to add the following to the debug container:
securityContext:
capabilities:
add:
- SYS_PTRACE
Then access is normal:
$ cat /proc/7/environ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binHOSTNAME=golang-distroless-58b6c5f455-v9zkvSSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crtKUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443KUBERNETES_PORT_443_TCP_PROTO=tcpKUBERNETES_PORT_443_TCP_PORT=443KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1KUBERNETES_SERVICE_HOST=10.43.0.1KUBERNETES_SERVICE_PORT=443KUBERNETES_SERVICE_PORT_HTTPS=443KUBERNETES_PORT=tcp://10.43.0.1:443HOME=/root
Similarly, we can access the file system of the process:
$ cd /proc/7/root $ ls app bin boot dev etc home lib lib64 proc root run sbin sys tmp usr var
Without modifying the basic image of the container, use pod.spec.shareProcessNamespace: true to add sys in the security configuration_ The ptrace feature gives the debug container full shell access to debug applications. However, modifying YAML and security configuration is only suitable for the test environment, which is not allowed in the production environment.
We need to use kubectl debug.
3. Kubectl debug
kubectl debug can perform different operations for different resources:
- Load: create a copy of the running Pod and modify some properties. For example, use a new version of tag in the copy.
- Load: add a temporary container for the running Pod (described below). Use the tools in the temporary container for debugging without restarting the Pod.
- Node: create a Pod on the node, run in the node's host namespace, and access the node's file system.
3.1 temporary containers
Since Kubernetes 1.18, you can use kubectl to add a temporary container for running pod s. This command is still in the alpha phase, so you need to "feature gate" Open in.
When creating a k3s cluster using k3d, open the EphemeralContainers feature:
$ k3d cluster create test --k3s-arg "--kube-apiserver-arg=feature-gates=EphemeralContainers=true"@
Then create a temporary container. After creation, you will directly enter the container:
$ kubectl debug golang-distroless-85c4896c45-rkjwn -it --image=ubuntu --image-pull-policy=IfNotPresent #Temporary container shell $ apt update && apt install -y curl $ curl localhost:8080 Hello world!
It is worth noting that the temporary container cannot share the process namespace with the original container:
$ ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 02:59 pts/0 00:00:00 bash root 3042 1 0 03:02 pts/0 00:00:00 ps -ef
You can attach the temporary container to the target container by adding the parameter -- target=[container]. Unlike pod.spec.shareProcessNamespace, the process with process number 1 is the process of the target container, and the process of the latter is the process of the infra container / pause:
$ kubectl debug golang-distroless-85c4896c45-rkjwn -it --image=ubuntu --image-pull-policy=IfNotPresent --target=golang-distroless-example
Note: the current version does not support deleting temporary containers. Please refer to issue , supported versions:
3.2 copy Pod and add container
In addition to adding temporary containers, another way is to create a copy of the Pod and add a container. Note that this is an ordinary container, not a temporary container. Notice that -- share processes is added here
$ kubectl debug golang-distroless-85c4896c45-rkjwn -it --image=ubuntu --image-pull-policy=IfNotPresent --share-processes --copy-to=golang-distroless-debug
Note that -- share processes is added here, and pod.spec.shareProcessNamespace=true will be automatically added:
$ kubectl get po golang-distroless-debug -o jsonpath='{.spec.shareProcessNamespace}'
true
Note: using kubectl debug for debugging does not automatically add sys to pod_ Ptrace security feature, which means that if the UID s used by the container are inconsistent, the process space cannot be accessed. By the time of issuance, Planned to be supported in 1.23.
summary
At present, all of the above are not suitable for use in the production environment, and can not be debugged without modifying the Pod definition.
It is expected that sys will be added to the debug function after kubernetes version 1.23_ Ptrace support. Then try again.
The article is unified in the official account of the cloud.
