I. deploy ETCD cluster
1.1 install ETCD
Etcd is a distributed key value storage system based on Raft, which is developed by CoreOS. It is often used in service discovery, shared configuration and concurrency control (such as leader election, distributed lock, etc.). kubernetes uses etcd to store all running data.
Etcd is a distributed key value storage system based on Raft, which is developed by CoreOS. It is often used in service discovery, shared configuration and concurrency control (such as leader election, distributed lock, etc.). kubernetes uses etcd to store all running data. 2 [root@k8smaster01 ~]# cd /opt/k8s/work 3 [root@k8smaster01 work]# wget https://github.com/coreos/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gz 4 [root@k8smaster01 work]# tar -xvf etcd-v3.3.13-linux-amd64.tar.gz
1.2 distribution of ETCD
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 4 do 5 echo ">>> ${master_ip}" 6 scp etcd-v3.3.13-linux-amd64/etcd* root@${master_ip}:/opt/k8s/bin 7 ssh root@${master_ip} "chmod +x /opt/k8s/bin/*" 8 done
1.3 create etcd certificate and key
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cat > etcd-csr.json <<EOF 3 { 4 "CN": "etcd", 5 "hosts": [ 6 "127.0.0.1", 7 "172.24.8.71", 8 "172.24.8.72", 9 "172.24.8.73" 10 ], 11 "key": { 12 "algo": "rsa", 13 "size": 2048 14 }, 15 "names": [ 16 { 17 "C": "CN", 18 "ST": "Shanghai", 19 "L": "Shanghai", 20 "O": "k8s", 21 "OU": "System" 22 } 23 ] 24 } 25 EOF 26 #Create a CA certificate request file for etcd
Interpretation:
The hosts field specifies the etcd node IP or domain name list authorized to use the certificate, in which all three node IP of the etcd cluster need to be listed.
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \ 3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json \ 4 -profile=kubernetes etcd-csr.json | cfssljson -bare etcd #Generate CA key (CA key. PEM) and certificate (ca.pem)
1.4 distribute certificate and private key
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 4 do 5 echo ">>> ${master_ip}" 6 ssh root@${master_ip} "mkdir -p /etc/etcd/cert" 7 scp etcd*.pem root@${master_ip}:/etc/etcd/cert/ 8 done
1.5 create etcd's systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# cat > etcd.service.template <<EOF 4 [Unit] 5 Description=Etcd Server 6 After=network.target 7 After=network-online.target 8 Wants=network-online.target 9 Documentation=https://github.com/coreos 10 11 [Service] 12 Type=notify 13 WorkingDirectory=${ETCD_DATA_DIR} 14 ExecStart=/opt/k8s/bin/etcd \\ 15 --data-dir=${ETCD_DATA_DIR} \\ 16 --wal-dir=${ETCD_WAL_DIR} \\ 17 --name=##MASTER_NAME## \\ 18 --cert-file=/etc/etcd/cert/etcd.pem \\ 19 --key-file=/etc/etcd/cert/etcd-key.pem \\ 20 --trusted-ca-file=/etc/kubernetes/cert/ca.pem \\ 21 --peer-cert-file=/etc/etcd/cert/etcd.pem \\ 22 --peer-key-file=/etc/etcd/cert/etcd-key.pem \\ 23 --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \\ 24 --peer-client-cert-auth \\ 25 --client-cert-auth \\ 26 --listen-peer-urls=https://##MASTER_IP##:2380 \\ 27 --initial-advertise-peer-urls=https://##MASTER_IP##:2380 \\ 28 --listen-client-urls=https://##MASTER_IP##:2379,http://127.0.0.1:2379 \\ 29 --advertise-client-urls=https://##MASTER_IP##:2379 \\ 30 --initial-cluster-token=etcd-cluster-0 \\ 31 --initial-cluster=${ETCD_NODES} \\ 32 --initial-cluster-state=new \\ 33 --auto-compaction-mode=periodic \\ 34 --auto-compaction-retention=1 \\ 35 --max-request-bytes=33554432 \\ 36 --quota-backend-bytes=6442450944 \\ 37 --heartbeat-interval=250 \\ 38 --election-timeout=2000 39 Restart=on-failure 40 RestartSec=5 41 LimitNOFILE=65536 42 43 [Install] 44 WantedBy=multi-user.target 45 EOF
Interpretation:
Working directory, -- data dir: Specifies that the working directory and data directory are ${ETCD_DATA_DIR}, which needs to be created before starting the service;
--Wal dir: Specifies the wal directory. In order to improve performance, SSD s or disks different from -- data dir are generally used;
--Name: Specifies the node name. When the -- initial cluster state value is new, the parameter value of - name must be in the -- initial cluster list;
--CERT file, -- key file: the certificate and private key used by etcd server when communicating with client;
--Trusted CA file: the CA certificate signing the client certificate, which is used to verify the client certificate;
--Peer cert file, -- peer key file: certificate and private key used by etcd to communicate with peer;
--Peer trusted CA file: the CA certificate signing the peer certificate, which is used to verify the peer certificate.
1.6 modify the corresponding address of systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for (( i=0; i < 3; i++ )) 4 do 5 sed -e "s/##MASTER_NAME##/${MASTER_NAMES[i]}/" -e "s/##MASTER_IP##/${MASTER_IPS[i]}/" etcd.service.template > etcd-${MASTER_IPS[i]}.service 6 done
1.7 distribute etcd systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 4 do 5 echo ">>> ${master_ip}" 6 scp etcd-${master_ip}.service root@${master_ip}:/etc/systemd/system/etcd.service 7 done
II. Start and verify
2.1 start ETCD
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 4 do 5 echo ">>> ${master_ip}" 6 ssh root@${master_ip} "mkdir -p ${ETCD_DATA_DIR} ${ETCD_WAL_DIR}" 7 ssh root@${master_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd " & 8 done
2.2 check ETCD startup
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 4 do 5 echo ">>> ${master_ip}" 6 ssh root@${master_ip} "systemctl status etcd|grep Active" 7 done
2.3 verify service status
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 4 do 5 echo ">>> ${master_ip}" 6 ETCDCTL_API=3 /opt/k8s/bin/etcdctl \ 7 --endpoints=https://${master_ip}:2379 \ 8 --cacert=/etc/kubernetes/cert/ca.pem \ 9 --cert=/etc/etcd/cert/etcd.pem \ 10 --key=/etc/etcd/cert/etcd-key.pem endpoint health 11 done
2.4 view the current leader of ETCD
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh 2 [root@k8smaster01 ~]# ETCDCTL_API=3 /opt/k8s/bin/etcdctl \ 3 -w table --cacert=/etc/kubernetes/cert/ca.pem \ 4 --cert=/etc/etcd/cert/etcd.pem \ 5 --key=/etc/etcd/cert/etcd-key.pem \ 6 --endpoints=${ETCD_ENDPOINTS} endpoint status
As shown above, the current leader of ETCD cluster is 172.24.8.71.