background

A 7-node k8s cluster and a master node's disk are abnormal, which often causes the etcd of the node to write to the card, and then drags down the etcd cluster. Therefore, it is considered to migrate the etcd of the node.

As shown in the figure above, the disk of the tstr501384 node is abnormal. To migrate the above etcd to the tstr501405a node, you need to expand the etcd cluster first and add the tstr501405a node (IP 10.233.130.47).

Operation steps

1. In a master node operation (example: tstr501382), execute etcdctl member add to join the tstr501405a node to the etcd cluster.

etcdctl   --endpoints=https://10.233.130.15:2379,https://10.233.130.16:2379,https://10.233.130.17:2379   --cacert /etc/kubernetes/pki/etcd/ca.crt    --cert /etc/kubernetes/pki/etcd/peer.crt   --key /etc/kubernetes/pki/etcd/peer.key member add  "tstr501405a"   --peer-urls="https://10.233.130.47:2380"

As shown in the figure above, the tstr501405a node has joined the etcd cluster, but it is in the unstarted phase.

2. Copy a master node's related files to the node to be added, mainly etcd.yaml and related certificates to start etcd static pod.

scp -rp  /etc/kubernetes/manifests/etcd.yaml  /etc/kubernetes/pki/  cloudops@10.233.130.47:/home/cloudops/addetcd

3. Operate on the etcd node to be added (for example, the / home/cloudops/addetcd directory of tstr501405a node), use the copied certificate, modify the relevant information, and make the certificate needed to start the new etcd node (mainly peer.crt and server.crt).

The following is the operation of making peer.crt certificate

#######Make a peer.crt certificate for the node to be added. Operate on the node to be added as follows, and change the dns and ip information to the corresponding node information

cat <<EOF>peer-ssl.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[v3_req]
keyUsage =critical, digitalSignature, keyEncipherment
extendedKeyUsage = TLS Web Server Authentication, TLS Web Client Authentication
subjectAltName = @alt_names
[alt_names]
DNS.1 = tstr501405a
DNS.2 = localhost
IP.1 = 10.233.130.47
IP.2 = 127.0.0.1
EOF

#####Use the copied master node certificate information to generate the peer.crt certificate of the node to be added
openssl req -new -key ./pki/etcd/peer.key -out peer.csr -subj "/CN=$HOSTNAME" -config peer-ssl.conf

openssl x509 -req -in peer.csr -CA ./pki/etcd/ca.crt -CAkey ./pki/etcd/ca.key -CAcreateserial -out peer.crt -days 3650 -extensions v3_req -extfile peer-ssl.conf

The following is the operation of creating server.crt certificate

#######Make the server.crt certificate of the node to be added. Operate on the node to be added as follows. Pay attention to modifying the dns and ip information as the corresponding node information

cat <<EOF>server-ssl.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[v3_req]
keyUsage =critical, digitalSignature, keyEncipherment
extendedKeyUsage = TLS Web Server Authentication, TLS Web Client Authentication
subjectAltName = @alt_names
[alt_names]
DNS.1 = tstr501405a
DNS.2 = localhost
IP.1 = 10.233.130.47
IP.2 = 127.0.0.1
EOF

openssl req -new -key ./pki/etcd/server.key -out server.csr -subj "/CN=$HOSTNAME" -config server-ssl.conf

openssl x509 -req -in server.csr -CA ./pki/etcd/ca.crt -CAkey ./pki/etcd/ca.key -CAcreateserial -out server.crt -days 3650 -extensions v3_req -extfile server-ssl.conf

4. Modify the etcd.yaml file copied from the master node with reference to the returned result when performing the etcdctl member add operation in step 1.

[root@tstr501405a addetcd]# cat etcd.yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ""
  creationTimestamp: null
  labels:
    component: etcd
    tier: control-plane
  name: etcd
  namespace: kube-system
spec:
  containers:
  - command:
    - etcd
    - --advertise-client-urls=https://10.233.130.47:2379
    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --client-cert-auth=true
    - --data-dir=/var/lib/etcd
    - --initial-advertise-peer-urls=https://10.233.130.47:2380
    - --initial-cluster=tstr501383=https://10.233.130.16:2380,tstr501405a=https://10.233.130.47:2380,tstr501384=https://10.233.130.17:2380,tstr501382=https://10.233.130.15:2380
    - --initial-cluster-state=existing
    - --key-file=/etc/kubernetes/pki/etcd/server.key
    - --listen-client-urls=https://127.0.0.1:2379,https://10.233.130.47:2379
    - --listen-peer-urls=https://10.233.130.47:2380
    - --name=tstr501405a
    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
    - --peer-client-cert-auth=true
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --snapshot-count=10000
    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    image: 10.233.71.70:60080/claas/etcd:3.2.24
    imagePullPolicy: IfNotPresent
    livenessProbe:
      exec:
        command:
        - /bin/sh
        - -ec
        - ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt
          --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key
          get foo

5. Copy related certificates to / etc/kubernetes/pki/etcd / directory (tstr501405a node / home/cloudops/addetcd directory operation), copy modified etcd.yaml to / etc/kubernetes/manifests /, wait for etcd static pod to start and confirm to join etcd cluster.

cp peer.crt  server.crt  /etc/kubernetes/pki/etcd/
cp ./pki/etcd/healthcheck-client.crt ./pki/etcd/healthcheck-client.key  ./pki/etcd/server.key ./pki/etcd/peer.key  ./pki/etcd/ca.crt    /etc/kubernetes/pki/etcd/
cp etcd.yaml /etc/kubernetes/manifests/

6. Confirm that the node etcd to be added starts successfully.

As shown in the figure above, confirm that the pod of node etcd of tstr501405a newly added to the etcd cluster is started successfully, and the status changes to started.