1.
Basic concepts
statefulset, which can be translated into stateful settings.
Contrast with deployment
The pod created by deployment deployment is stateless. The rescheduling of pod, the name of pod hostname, the order of starting pod and deleting pod are random. The deployment uses shared storage, and all pods share a single volume.
The pod created by the stateful set deployment is stateful. Rescheduling the pod, the name of the pod host remains unchanged. Starting the pod sequence to delete the pod sequence can perform operations in an orderly manner according to the defined order. Orderly dynamic updates. The storage used by the stateful set is not a shared storage volume. A pod corresponds to a storage volume (pv).pod is re-organized. Create scheduled mounted storage volumes that remain unchanged.
2.
Technical concepts used in statefulset implementation
headless service
headless service does not need to configure cluster IP, and each pod will have a corresponding dns domain name. Therefore, resolvable dns records can be generated for each pod.
statefulset
statefulset is used to deploy and manage pod resources
volumeClaimTemplates
Implementing a pv storage volume for each pod
3.
Test environment preparation
The kubernets cluster must deploy dns, which can be parsed properly.
The kubernetes cluster must configure storage class to dynamically create pv.
I use nfs storage environment here.
4.
Analytical comparison of headless service and service
service
[root@k8s-master1 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE httpd-svc NodePort 10.254.33.250 <none> 80:8768/TCP 92d kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 92d
Note the following parsing of the ip used for dns
[root@k8s-master1 ~]# kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.254.0.2 <none> 53/UDP,53/TCP 92d
[root@k8s-master1 ~]# kubectl describe svc httpd-svc
Name: httpd-svc
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"httpd-svc","namespace":"default"},"spec":{"ports":[{"port":80}],"selector":{"a...
Selector: app=httpd-app
Type: NodePort
IP: 10.254.33.250
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 8768/TCP
Endpoints: 172.30.37.7:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
[root@k8s-master1 ~]#pod with default name space is parsed by nslookup
[root@k8s-master1 ~]# kubectl exec -it nfs-client-provisioner-65bf6bd464-qdzcj nslookup httpd-svc.default.svc.cluster.local 10.254.0.2 Server: 10.254.0.2 Address 1: 10.254.0.2 kube-dns.kube-system.svc.cluster.local Name: httpd-svc.default.svc.cluster.local Address 1: 10.254.33.250 httpd-svc.default.svc.cluster.local [root@k8s-master1 ~]#
headless service
[root@k8s-master1 test]# cat headless-svc-test.yaml apiVersion: v1 kind: Service metadata: name: headless-svc labels: app: headless-svc spec: ports: - port: 80 name: myweb selector: app: headless-pod clusterIP: None --- apiVersion: apps/v1 kind: Deployment metadata: name: headless-test spec: replicas: 3 selector: matchLabels: app: headless-pod template: metadata: labels: app: headless-pod spec: containers: - image: httpd name: myhttpd ports: - containerPort: 80 [root@k8s-master1 test]#
[root@k8s-master1 test]# kubectl describe svc headless-svc
Name: headless-svc
Namespace: default
Labels: app=headless-svc
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"headless-svc"},"name":"headless-svc","namespace":"default"},"spec":{"...
Selector: app=headless-pod
Type: ClusterIP
IP: None
Port: myweb 80/TCP
TargetPort: 80/TCP
Endpoints: 172.30.65.4:80,172.30.65.5:80,172.30.81.6:80
Session Affinity: None
Events: <none>
[root@k8s-master1 test]#pod with default name space is parsed by nslookup
[root@k8s-master1 test]# kubectl exec -it nfs-client-provisioner-65bf6bd464-qdzcj nslookup headless-svc.default.svc.cluster.local 10.254.0.2 Server: 10.254.0.2 Address 1: 10.254.0.2 kube-dns.kube-system.svc.cluster.local Name: headless-svc.default.svc.cluster.local Address 1: 172.30.65.5 172-30-65-5.headless-svc.default.svc.cluster.local Address 2: 172.30.65.4 172-30-65-4.headless-svc.default.svc.cluster.local Address 3: 172.30.81.6 172-30-81-6.headless-svc.default.svc.cluster.local [root@k8s-master1 test]#
The comparison shows that:
The difference between configuring headless service and service is simply to add a cluster IP: None
The difference of dns analysis
service parsing returns ip and dns domain names of svc
headless service parses ip and dns domain names returned to pod
headless service parsing returns the hostname of the pod.
The stateful set requires a fixed pod name,hostname, and must be able to parse the host name of the pod. Therefore, headless service is used.
5.
statefulset configuration test
[root@k8s-master1 test]# cat statefulset-test.yaml apiVersion: v1 kind: Service metadata: name: headless-svc labels: app: headless-svc spec: ports: - port: 80 name: myweb selector: app: headless-pod clusterIP: None --- apiVersion: apps/v1 kind: StatefulSet metadata: name: statefulset-test spec: serviceName: headless-svc replicas: 3 selector: matchLabels: app: headless-pod template: metadata: labels: app: headless-pod spec: containers: - image: httpd name: myhttpd ports: - containerPort: 80 name: httpd [root@k8s-master1 test]#
[root@k8s-master1 test]# kubectl describe svc headless-svc
Name: headless-svc
Namespace: default
Labels: app=headless-svc
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"headless-svc"},"name":"headless-svc","namespace":"default"},"spec":{"...
Selector: app=headless-pod
Type: ClusterIP
IP: None
Port: myweb 80/TCP
TargetPort: 80/TCP
Endpoints: 172.30.65.4:80,172.30.65.5:80,172.30.81.6:80
Session Affinity: None
Events: <none>[root@k8s-master1 test]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE statefulset-test-0 1/1 Running 0 4m 172.30.65.4 k8s-master3 statefulset-test-1 1/1 Running 0 4m 172.30.81.6 k8s-master2 statefulset-test-2 1/1 Running 0 4m 172.30.65.5 k8s-master3
hostname [root@k8s-master1 test]# kubectl exec -it statefulset-test-0 hostname statefulset-test-0 [root@k8s-master1 test]# kubectl exec -it statefulset-test-1 hostname statefulset-test-1 [root@k8s-master1 test]# kubectl exec -it statefulset-test-2 hostname statefulset-test-2
analysis
[root@k8s-master1 test]# kubectl exec -it nfs-client-provisioner-65bf6bd464-qdzcj nslookup headless-svc.default.svc.cluster.local 10.254.0.2 Server: 10.254.0.2 Address 1: 10.254.0.2 kube-dns.kube-system.svc.cluster.local Name: headless-svc.default.svc.cluster.local Address 1: 172.30.65.5 statefulset-test-2.headless-svc.default.svc.cluster.local Address 2: 172.30.65.4 statefulset-test-0.headless-svc.default.svc.cluster.local Address 3: 172.30.81.6 statefulset-test-1.headless-svc.default.svc.cluster.local [root@k8s-master1 test]#
statefulset deployment versus head services deployment deployment
The pod name is generated sequentially, starting at 0 and naming the pod at + 1 each time.
Creating pod s is also created one by one in order, not in batches and all at once as deployment deployment deployment deployment deployment deployment does.
test
Delete pod
[root@k8s-master1 test]# kubectl delete pod statefulset-test-0 pod "statefulset-test-0" deleted
[root@k8s-master1 test]# kubectl get pod NAME READY STATUS RESTARTS AGE statefulset-test-0 0/1 Terminating 0 19m statefulset-test-1 1/1 Running 0 19m statefulset-test-2 1/1 Running 0 19m
Re-create pod automatically
[root@k8s-master1 test]# kubectl get pod NAME READY STATUS RESTARTS AGE statefulset-test-0 0/1 ContainerCreating 0 0s statefulset-test-1 1/1 Running 0 19m statefulset-test-2 1/1 Running 0 19m
[root@k8s-master1 test]# kubectl get pod NAME READY STATUS RESTARTS AGE statefulset-test-0 1/1 Running 0 4s statefulset-test-1 1/1 Running 0 20m statefulset-test-2 1/1 Running 0 19m
[root@k8s-master1 test]# kubectl exec -it statefulset-test-0 hostname statefulset-test-0 [root@k8s-master1 test]#
You can see that the name of the pod and hostname have not changed since the pod was recreated.
Change replicas to 1
[root@k8s-master1 test]# kubectl apply -f statefulset-test.yaml service "headless-svc" unchanged statefulset.apps "statefulset-test" configured [root@k8s-master1 test]# kubectl get pod NAME READY STATUS RESTARTS AGE statefulset-test-0 1/1 Running 0 4m statefulset-test-1 1/1 Running 0 24m statefulset-test-2 0/1 Terminating 0 24m [root@k8s-master1 test]# kubectl get pod NAME READY STATUS RESTARTS AGE statefulset-test-0 1/1 Running 0 4m statefulset-test-1 0/1 Terminating 0 24m [root@k8s-master1 test]# kubectl get pod NAME READY STATUS RESTARTS AGE statefulset-test-0 1/1 Running 0 5m
The pod is deleted one by one in order
Change replicas back to 3
[root@k8s-master1 test]# kubectl apply -f statefulset-test.yaml service "headless-svc" unchanged statefulset.apps "statefulset-test" configured [root@k8s-master1 test]# kubectl get pod NAME READY STATUS RESTARTS AGE statefulset-test-0 1/1 Running 0 6m statefulset-test-1 0/1 ContainerCreating 0 4s [root@k8s-master1 test]# kubectl get pod NAME READY STATUS RESTARTS AGE statefulset-test-0 1/1 Running 0 6m statefulset-test-1 1/1 Running 0 8s statefulset-test-2 0/1 ContainerCreating 0 1s [root@k8s-master1 test]# kubectl get pod NAME READY STATUS RESTARTS AGE statefulset-test-0 1/1 Running 0 6m statefulset-test-1 1/1 Running 0 14s statefulset-test-2 1/1 Running 0 7s [root@k8s-master1 test]# [root@k8s-master1 test]# kubectl exec -it statefulset-test-2 hostname statefulset-test-2 [root@k8s-master1 test]#
The pod is created one by one in order, and the pod name and hostname remain unchanged
6.
statefulset combined with volume ClaimTemplates test
[root@k8s-master1 test]# cat statefulset-test.yaml apiVersion: v1 kind: Service metadata: name: headless-svc labels: app: headless-svc spec: ports: - port: 80 name: myweb selector: app: headless-pod clusterIP: None --- apiVersion: apps/v1 kind: StatefulSet metadata: name: statefulset-test spec: serviceName: headless-svc replicas: 3 selector: matchLabels: app: headless-pod template: metadata: labels: app: headless-pod spec: containers: - image: httpd name: myhttpd ports: - containerPort: 80 name: httpd volumeMounts: - mountPath: /mnt name: test volumeClaimTemplates: - metadata: name: test annotations: volume.beta.kubernetes.io/storage-class: managed-nfs-storage spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 100Mi
Pay attention to the format. Multiple spaces and fewer spaces will fail.
annotations: volume.beta.kubernetes.io/storage-class: managed-nfs-storage
This is to specify storage class
[root@k8s-master1 test]# kubectl get pv,pvc NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-a1c6928d-aebc-11e9-8580-000c291d7023 100Mi RWO Delete Bound default/test-statefulset-test-0 managed-nfs-storage 1m persistentvolume/pvc-a94813df-aebc-11e9-8580-000c291d7023 100Mi RWO Delete Bound default/test-statefulset-test-1 managed-nfs-storage 1m persistentvolume/pvc-b234f5a7-aebc-11e9-8580-000c291d7023 100Mi RWO Delete Bound default/test-statefulset-test-2 managed-nfs-storage 49s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/test-statefulset-test-0 Bound pvc-a1c6928d-aebc-11e9-8580-000c291d7023 100Mi RWO managed-nfs-storage 1m persistentvolumeclaim/test-statefulset-test-1 Bound pvc-a94813df-aebc-11e9-8580-000c291d7023 100Mi RWO managed-nfs-storage 1m persistentvolumeclaim/test-statefulset-test-2 Bound pvc-b234f5a7-aebc-11e9-8580-000c291d7023 100Mi RWO managed-nfs-storage 1m [root@k8s-master1 test]#
Retrieve the pv volume corresponding to pod and intercept volume for comparison
[root@k8s-master1 test]# kubectl describe pod statefulset-test-0 Name: statefulset-test-0 Namespace: default Volumes: test: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test-statefulset-test-0 ReadOnly: false [root@k8s-master1 test]# kubectl describe pod statefulset-test-1 Name: statefulset-test-1 Namespace: default Volumes: test: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test-statefulset-test-1 ReadOnly: false [root@k8s-master1 test]# kubectl describe pod statefulset-test-2 Name: statefulset-test-2 Namespace: default Volumes: test: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test-statefulset-test-2 ReadOnly: false
You can see that a pod corresponds to a pv volume.
Retrieving the nfs server access directory, you can see the new pvc directory
[root@k8s-master3 k8s]# ls default-test-statefulset-test-0-pvc-a1c6928d-aebc-11e9-8580-000c291d7023 default-test-statefulset-test-1-pvc-a94813df-aebc-11e9-8580-000c291d7023 default-test-statefulset-test-2-pvc-b234f5a7-aebc-11e9-8580-000c291d7023 [root@k8s-master3 k8s]#
test
Delete the pod and see if you can re-correspond to the original pvc volume?
[root@k8s-master1 test]# kubectl get pod NAME READY STATUS RESTARTS AGE statefulset-test-0 1/1 Running 0 12m statefulset-test-1 1/1 Running 0 11m statefulset-test-2 1/1 Running 0 11m
Create a new file in statefulset-test-1 and statefulset-test-2 and store it in pvc
[root@k8s-master1 test]# kubectl exec -it statefulset-test-1 touch /mnt/t1 [root@k8s-master1 test]# kubectl exec -it statefulset-test-2 touch /mnt/t2
Retrieval of pvc volumes
[root@k8s-master3 k8s]# cd default-test-statefulset-test-1-pvc-a94813df-aebc-11e9-8580-000c291d7023/ [root@k8s-master3 default-test-statefulset-test-1-pvc-a94813df-aebc-11e9-8580-000c291d7023]# ls t1 [root@k8s-master3 default-test-statefulset-test-1-pvc-a94813df-aebc-11e9-8580-000c291d7023]# cd .. [root@k8s-master3 k8s]# cd default-test-statefulset-test-2-pvc-b234f5a7-aebc-11e9-8580-000c291d7023/ [root@k8s-master3 default-test-statefulset-test-2-pvc-b234f5a7-aebc-11e9-8580-000c291d7023]# ls t2
Delete the two pod s
[root@k8s-master1 test]# kubectl delete pod statefulset-test-1 statefulset-test-2 pod "statefulset-test-1" deleted pod "statefulset-test-2" deleted
After deletion, pod is created automatically
Retrieve the recreated pod
[root@k8s-master1 test]# kubectl describe pod statefulset-test-1 Name: statefulset-test-1 Namespace: default Volumes: test: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test-statefulset-test-1 ReadOnly: false [root@k8s-master1 test]# kubectl describe pod statefulset-test-2 Name: statefulset-test-2 Namespace: default Volumes: test: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test-statefulset-test-2 ReadOnly: false
You can see that the pod command hostname and storage volume are all kept the same before deletion.
[root@k8s-master1 test]# kubectl exec -it statefulset-test-1 ls /mnt t1 [root@k8s-master1 test]# kubectl exec -it statefulset-test-2 ls /mnt t2 [root@k8s-master1 test]#
The files created earlier that hold the volume remain unchanged.
7.
One question left behind is:
Deployment deployment, svc accesses back-end pod through kube-proxy combined with ipvs and iptables.
statefulset deployment, how does svc achieve access to back-end pod? What are the rules?