Powerful self-healing capability is an important feature of container orchestration engines such as Kubernetes. The default way to implement self-healing is to automatically rekindle the failing container. In addition, users can use Liveness and Readines detection mechanism to set up more detailed health checks, and then achieve the following requirements:
- Zero downtime deployment.
- Avoid deploying invalid mirrors.
- More secure rolling upgrades.
Next, learn Kubernetes'Helth Check function through practice.
Default health check
First, learn Kubernetes default health screening mechanism:
Each container starts with a process specified by the Dockerfile CMD or ENTRYPOINT. If the return code is not zero when the process exits, Kubernetes will restart the container according to restart policy if the container fails.
Let's simulate a container failure scenario. The Pod configuration file is as follows:
apiVersion: v1
kind: Pod
metadata:
labels:
test: healthcheck
name: healthcheck
spec:
restartPolicy: OnFailure
containers:
- name: healthcheck
image: busybox
args:
- /bin/sh
- -c
- sleep 10; exit 1Pod's restart policy is set to OnFailure by default to Always.
sleep 10; exit 1 simulates a container failure after 10 seconds of startup.
Execute kubectl application to create a Pod named healthcheck.
# kubectl apply -f healthcheck.yml pod/healthcheck created
Look at the status of the Pod in a few minutes:
# kubectl get pod healthcheck NAME READY STATUS RESTARTS AGE healthcheck 0/1 CrashLoopBackOff 4 3m39s
You can see that the container has been restarted four times.
In the example above, the return value of the container process is non-zero, while Kubernetes believes that the container has failed and needs to be restarted. However, there are many cases where failures occur, but the process will not quit. For example, when accessing the Web server, 500 internal errors may be system overload or resource deadlock. At this time, the httpd process does not exit abnormally. In this case, restarting the container may be the most direct and effective solution. How can we use the Health Check mechanism to deal with such scenarios?
The answer is Liveness detection.
Liveness detection
Liveness probes allow users to customize the conditions under which containers are healthy. If the probe fails, Kubernetes restarts the container.
Or as an example, create the following Pod:
Start the process by creating the file / TMP / health first and deleting it after 30 seconds. In our settings, if the file / TMP / health exists, the container is assumed to be in normal condition, otherwise it will fail.
The liveness Probe section defines how to perform Liveness probes:
- The method of detection is to check whether the / TMP / health file exists through the cat command. If the command is successfully executed and the return value is zero, Kubernetes considers the Liveness probe to be successful; if the return value of the command is non-zero, the Liveness probe fails.
- Initial DelaySeconds: 10 Specifies that the container will start executing Liveness probes after starting 10, and we usually set it according to the application's startup preparation time. For example, if it takes 30 seconds for an application to start normally, the initial DelaySeconds value should be greater than 30.
- PeriododSeconds: 5 specifies that Liveness probes are performed every five seconds. If Kubernetes fails to perform three Liveness probes in a row, the container will be killed and restarted.
Create Pod liveness below:
# kubectl apply -f liveness.yaml pod/liveness created
From the configuration file, in the first 30 seconds, / TMP / health exists, the cat command returns 0, and the Liveness probe succeeds. During this period, the Events section of kubectl description pod liveness displays the normal log.
# kubectl describe pod liveness
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 31s default-scheduler Successfully assigned default/liveness to k8s-node2 Normal Pulling 30s kubelet, k8s-node2 Pulling image "busybox" Normal Pulled 30s kubelet, k8s-node2 Successfully pulled image "busybox" Normal Created 30s kubelet, k8s-node2 Created container liveness Normal Started 29s kubelet, k8s-node2 Started container liveness
After 35 seconds, the log will show that / TMP / health no longer exists and that Liveness detection failed. In a few tens of seconds, after several failed probes, the container will be restarted.
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 47s default-scheduler Successfully assigned default/liveness to k8s-node2 Normal Pulling 46s kubelet, k8s-node2 Pulling image "busybox" Normal Pulled 46s kubelet, k8s-node2 Successfully pulled image "busybox" Normal Created 46s kubelet, k8s-node2 Created container liveness Normal Started 45s kubelet, k8s-node2 Started container liveness Warning Unhealthy 3s (x3 over 13s) kubelet, k8s-node2 Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory Normal Killing 3s kubelet, k8s-node2 Container liveness failed liveness probe, will be restarted
# kubectl get pod liveness NAME READY STATUS RESTARTS AGE liveness 1/1 Running 1 76s
In addition to Liveness detection, the Kubernetes Health Check mechanism also includes Readines detection.
Readines Detection
Users can tell Kubernetes when to self-heal by restarting containers through Liveness detection; Readiness detection tells Kubernetes when to add containers to Service load balancing pool.
The configuration syntax of Readiness probes is exactly the same as Liveness probes. Here's an example:
apiVersion: v1
kind: Pod
metadata:
labels:
test: readiness
name: readiness
spec:
restartPolicy: OnFailure
containers:
- name: readiness
image: busybox
args:
- /bin/sh
- -c
- touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
readinessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 10
periodSeconds: 5This configuration file just replaces liveness in the previous example with readiness, so let's see what's different.
[root@k8s-master ~]# kubectl get pod readiness NAME READY STATUS RESTARTS AGE readiness 0/1 Running 0 10s [root@k8s-master ~]# kubectl get pod readiness NAME READY STATUS RESTARTS AGE readiness 1/1 Running 0 20s [root@k8s-master ~]# kubectl get pod readiness NAME READY STATUS RESTARTS AGE readiness 1/1 Running 0 35s [root@k8s-master ~]# kubectl get pod readiness NAME READY STATUS RESTARTS AGE readiness 0/1 Running 0 61s [root@k8s-master ~]# kubectl describe pod readiness
The READY state of Pod readiness has undergone the following changes:
- When it was first created, the READY state was unavailable.
- Fifteen seconds later (initial DelaySeconds + periodSeconds), the first Readines probe was performed and successfully returned, setting READY as available.
- Thirty seconds later, / TMP / health was deleted and READY was set to be unavailable after three successive failed Readines probes.
You can also see the log of Readines detection failure through kubectl description pod Readiness.
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 95s default-scheduler Successfully assigned default/readiness to k8s-node2 Normal Pulling 94s kubelet, k8s-node2 Pulling image "busybox" Normal Pulled 94s kubelet, k8s-node2 Successfully pulled image "busybox" Normal Created 93s kubelet, k8s-node2 Created container readiness Normal Started 93s kubelet, k8s-node2 Started container readiness Warning Unhealthy 4s (x12 over 59s) kubelet, k8s-node2 Readiness probe failed: cat: can't open '/tmp/healthy': No such file or directory
Here's a comparison between Liveness detection and Readiness detection:
- Liveness detection and Readiness detection are two Health Check mechanisms. If not specifically configured, Kubernetes will adopt the same default behavior for the two detections, that is, to judge whether the detection is successful by determining whether the return value of the container start process is zero.
- The configuration methods of the two probes are exactly the same, and the configuration parameters supported are the same. The difference lies in the behavior after detection failure: Liveness detection is to restart the container; Readiness detection is to set the container unavailable and not receive requests forwarded by Service.
- Liveness probes and Readiness probes are executed independently and do not depend on each other, so they can be used separately or simultaneously. Liveness detection is used to determine whether the container needs to be restarted to achieve self-healing; Readiness detection is used to determine whether the container is ready to provide services to the outside world.
Use Health Check in business scenarios.
Use Health Check in Rolling Update
The previous section discussed the application of Health Check in Scale Up. Another important application scenario of Health Check is Rolling Update. Imagine the following:
There is a functioning multi-copy application, and then the application is updated (for example, using a higher version of image), Kubernetes will start the new copy, and the following events occur:
- Normally, a new copy takes 10 seconds to prepare and cannot respond to business requests until then.
- However, due to human configuration errors, replicas are always unable to complete the preparatory work (for example, unable to connect to the back-end database).
Think about it: What happens if you don't configure Health Check?
Because the new replica itself does not exit abnormally, the default Halith Check mechanism assumes that the container is ready and will gradually replace the existing replica with the new replica. As a result, when all the old replicas are replaced, the entire application will not be able to process requests and provide services to the outside world. If this happens in an important production system, the consequences will be very serious.
If Health Check is configured correctly, the new copy will be added to Service only if it passes Readines detection; if it does not pass detection, the existing copy will not be completely replaced, and the business will still be normal.
Here is an example to practice the application of Health Check in Rolling Update.
Simulate a 10-copy application with the following configuration file app.v1.yml:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: app
spec:
replicas: 10
template:
metadata:
labels:
run: app
spec:
containers:
- name: app
image: busybox
args:
- /bin/sh
- -c
- sleep 3000
readinessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 10
periodSeconds: 5After 10 seconds, the copy can be detected by Readines.
# kubectl get deployments. app NAME READY UP-TO-DATE AVAILABLE AGE app 0/10 10 0 8s # kubectl get pod NAME READY STATUS RESTARTS AGE app-6dd7f876c4-575v5 1/1 Running 0 25s app-6dd7f876c4-9kwk9 1/1 Running 0 25s app-6dd7f876c4-bx4pf 1/1 Running 0 25s app-6dd7f876c4-f6qf2 1/1 Running 0 25s app-6dd7f876c4-fxp2m 1/1 Running 0 25s app-6dd7f876c4-k76mr 1/1 Running 0 25s app-6dd7f876c4-mfqsq 1/1 Running 0 25s app-6dd7f876c4-whkc7 1/1 Running 0 25s app-6dd7f876c4-x9q87 1/1 Running 0 25s app-6dd7f876c4-xf8dv 1/1 Running 0 25s
Next, scroll to update the application. The configuration file app.v2.yml is as follows:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: app
spec:
replicas: 10
template:
metadata:
labels:
run: app
spec:
containers:
- name: app
image: busybox
args:
- /bin/sh
- -c
- sleep 3000
readinessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 10
periodSeconds: 5Obviously, because / TMP / health does not exist in the new copy, it cannot be detected by Readines. Verification is as follows:
# kubectl apply -f app.yml --record deployment.apps/app configured [root@k8s-master ~]# kubectl get deployments. app NAME READY UP-TO-DATE AVAILABLE AGE app 8/10 5 8 80s # kubectl get pod NAME READY STATUS RESTARTS AGE app-6dd7f876c4-575v5 1/1 Running 0 2m3s app-6dd7f876c4-9kwk9 1/1 Running 0 2m3s app-6dd7f876c4-f6qf2 1/1 Running 0 2m3s app-6dd7f876c4-fxp2m 1/1 Running 0 2m3s app-6dd7f876c4-k76mr 1/1 Running 0 2m3s app-6dd7f876c4-whkc7 1/1 Running 0 2m3s app-6dd7f876c4-x9q87 1/1 Running 0 2m3s app-6dd7f876c4-xf8dv 1/1 Running 0 2m3s app-7d7559dd99-6w2kn 0/1 Running 0 49s app-7d7559dd99-jnbxg 0/1 Running 0 49s app-7d7559dd99-mxbwg 0/1 Running 0 49s app-7d7559dd99-n59vq 0/1 Running 0 49s app-7d7559dd99-t49cp 0/1 Running 0 49s
This screenshot contains a lot of information and deserves our detailed analysis.
Focus first on the kubectl get pod output:
- Judging from Pod's AGE column, the last five Pods are new copies and are currently in the NOT READY state.
- Old copies were reduced from 10 to 8.
Look again at the output of kubectl get deployment app:
- DESIRED 10 indicates that the desired state is 10 copies of READY.
- CURRENT 13 represents the total number of current copies: eight old copies + five new copies.
- UP-TO-DATE 5 indicates the current number of copies that have been updated: five new copies.
- AVAILABLE 8 represents the number of copies currently in READY status: eight old copies.
In our settings, new copies are never detectable through Readines, so this state will remain unchanged.
Above we simulated a scenario where rolling updates failed. Fortunately, however, Health Check helped us block defective copies, while retaining most of the old copies, and the business was not affected by the failure of the update.
Next we have to answer: Why are five new copies created and only two old ones destroyed?
The reason is that rolling updates control the number of replica substitutions by parameters maxSurge and maxUnavailable.
maxSurge
This parameter controls the number of replicas exceeding the upper limit of DESIRED during rolling updates. MaxSurge can be a specific integer (such as 3), or it can be 100 percent, rounded up. The default value of maxSurge is 25%.
In the example above, DESIRED is 10, and the maximum number of copies is:
roundUp(10 + 10 * 25%) = 13
So we see that CURRENT is 13.
maxUnavailable
This parameter controls the maximum proportion of unavailable copies in DESIRED during rolling updates. MaxUnavailable can be a specific integer (for example, 3), or it can be a hundred percent, rounding down. The default value of maxUnavailable is 25%.
In the example above, DESIRED is 10, so the number of copies available is at least:
10 - roundDown(10 * 25%) = 8
So we see AVAILABLE is 8.
The larger the maxSurge value, the more new copies are initially created; the larger the maxUnavailable value, the more old copies are initially destroyed.
Ideally, the process of rolling updates in our case should be as follows:
- First, three new copies are created to bring the total number of copies to 13.
- Then destroy two old copies to reduce the number of available copies to eight.
- When these two old copies are destroyed successfully, two new copies can be created to keep the total number of copies to 13.
- When a new copy is detected by Readines, the number of available copies increases to more than 8.
- Further more old copies can be destroyed, bringing the number of available copies back to 8.
- The destruction of old copies makes the total number of copies less than 13, which allows more new copies to be created.
- This process will continue, and eventually all the old copies will be replaced by the new ones, and the rolling updates will be completed.
Our actual situation is that in Step 4, we got stuck and the new copy could not be detected by Readines. This process can be viewed in the log section of the kubectl describe deployment app.
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 11m deployment-controller Scaled up replica set app-6dd7f876c4 to 10 Normal ScalingReplicaSet 10m deployment-controller Scaled up replica set app-7d7559dd99 to 3 Normal ScalingReplicaSet 10m deployment-controller Scaled down replica set app-6dd7f876c4 to 8 Normal ScalingReplicaSet 10m deployment-controller Scaled up replica set app-7d7559dd99 to 5
If the scroll update fails, you can roll back to the previous version through kubectl rollout undo.
# kubectl rollout history deployment app deployment.extensions/app REVISION CHANGE-CAUSE 1 kubectl apply --filename=app.yml --record=true 2 kubectl apply --filename=app.yml --record=true
# kubectl get deployments. app NAME READY UP-TO-DATE AVAILABLE AGE app 8/10 5 8 14m
kubectl get pod NAME READY STATUS RESTARTS AGE app-6dd7f876c4-575v5 1/1 Running 0 14m app-6dd7f876c4-9kwk9 1/1 Running 0 14m app-6dd7f876c4-f6qf2 1/1 Running 0 14m app-6dd7f876c4-fxp2m 1/1 Running 0 14m app-6dd7f876c4-k76mr 1/1 Running 0 14m app-6dd7f876c4-whkc7 1/1 Running 0 14m app-6dd7f876c4-x9q87 1/1 Running 0 14m app-6dd7f876c4-xf8dv 1/1 Running 0 14m app-7d7559dd99-6w2kn 0/1 Running 0 13m app-7d7559dd99-jnbxg 0/1 Running 0 13m app-7d7559dd99-mxbwg 0/1 Running 0 13m app-7d7559dd99-n59vq 0/1 Running 0 13m app-7d7559dd99-t49cp 0/1 Running 0 13m
If you want to customize maxSurge and maxUnavailable, you can configure it as follows:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: app
spec:
strategy:
rollingUpdate:
maxSurge: 35%
maxUnavailable: 35%
replicas: 10
template:
metadata:
labels:
run: app
spec:
containers:
- name: app
image: busybox
args:
- /bin/sh
- -c
- sleep 3000
readinessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 10
periodSeconds: 5Summary
This chapter discusses two mechanisms of Kubernetes health examination: Liveness detection and Readines detection, and practices the application of health examination in Scale Up and Rolling Update scenarios.