Kubedge deployment ~ Kubernetes 1.17.4 deployment

Keywords: kubelet Kubernetes Docker network

This paper introduces how to deploy Kubernetes 1.17.4 cluster using kubeadm on ubuntu 18.04 64 bit dual core CPU virtual machine. The network plug-in is flannel v0.11.0, and the image source is alicloud

1, Install docker

apt-get install docker.io
Execute the following command to create a new / etc/docker/daemon.json file:

cat > /etc/docker/daemon.json <<-EOF
  "registry-mirrors": [
  "exec-opts": ["native.cgroupdriver=systemd"]
Registry mirrors is the address of the image accelerator.
native.cgroupdriver=systemd indicates that the cgroup driver used is systemd (k8s uses this method), and the default is cgroupfs. The reason for the modification is that the driver mode of k8s modified in kubeadm.conf is not successful.

Restart docker and view cgroup:

# systemctl restart docker 
# docker info | grep -i cgroup
Cgroup Driver: systemd
If systemd appears, the modification is successful.

2, Deploy k8s master host

k8s is deployed by master and node nodes. This section is the master host.

2.1 turn off swap

Edit the / etc/fstab file and comment out the lines attached by the swap partition. For example:

# swap was on /dev/sda5 during installation
UUID=aaa38da3-6e60-4e9d-bfc6-7128fd05f1c7 none swapsw  0  0

Execute again: ා swapoff -a

2.2 add domestic k8s source. Select Alibaba cloud's:

# cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
Add key: ා cat https://packages.cloud.google.com/apt/doc/apt-key.gpg| sudo apt key add-

If it is not successful, download it through some methods: https://packages.cloud.google.com/apt/doc/apt-key.gpg, and put it into the project directory. Re implementation:

# cat apt-key.gpg | sudo apt-key add -

3.3 update source

# apt-get update
Install kubeadm, kubectl, kubelet, kubernetes CNI and other tools.

# apt-get install -y kubeadm kubectl kubelet kubernetes-cni

Note 1: installation of kubeadm will automatically install kubectl, kubelet and kubernetes CNI, so it is OK to specify only kubeadm.
Note 2: when this article is installed, the version is 1.17.4 and kubernetes CNI is 0.7.5. The download file is located in the / var/cache/apt/archives / directory.

3.4 get the image version required for deployment

# kubeadm config images list
The output is as follows:

W1214 08:46:14.303772    8461 version.go:102] falling back to the local client version: v1.17.4
W1214 08:46:14.304223    8461 validation.go:28] Cannot validate kube-proxy config - no validator is available
W1214 08:46:14.304609    8461 validation.go:28] Cannot validate kubelet config - no validator is available

The warning message prompted above can be ignored. Here is the version of the image to confirm that this version of kubeadm matches, because compatibility problems may occur due to different component versions.

3.5 pull the image file.
Generally, the image of k8s.gcr.io cannot be downloaded directly in China. There are two ways:
1. When initializing k8s, use the Alibaba cloud image address, which can be downloaded successfully. See initialization below.
2. Download the above image. Use the following script pullk8s.sh (note that the script must add the x attribute):

# The following image should be removed from the prefix of "k8s.gcr.io /", and the version should be changed to the version obtained by the kubeadm config images list command

for imageName in ${images[@]} ; do
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
    docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName


chmod +x pullk8s.sh
bash pullk8s.sh (or. / pullk8s.sh)

3.6 network
Set network configuration:

mkdir -p /etc/cni/net.d

cat >/etc/cni/net.d/10-mynet.conf <<-EOF
    "cniVersion": "0.3.0",
    "name": "mynet",
    "type": "bridge",
    "bridge": "cni0",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
        "type": "host-local",
        "subnet": "",
        "routes": [
            {"dst": ""}

cat >/etc/cni/net.d/99-loopback.conf <<-EOF
    "cniVersion": "0.3.0",
    "type": "loopback"

After practice, this step is not OK.

3.7 download the flannel image
docker pull quay.io/coreos/flannel:v0.11.0-amd64
Note: if you cannot download, you need to use other methods.
flannel image information:

# docker images | grep flannel
quay.io/coreos/flannel   v0.11.0-amd64 ff281650a721 11 months ago 52.6MB

3.8 initialization
Version 1:

kubeadm init --pod-network-cidr= \
  --image-repository registry.aliyuncs.com/google_containers

– pod network CIDR specifies the network segment, which will be used by subsequent network plug-ins (flannel is used in this article).
– image repository specifies the image address, which is k8s.gcr.io by default, and Alibaba cloud image address registry.aliyuncs.com/google'u containers.
Note that other parameters default.

The above command is equivalent to the following command:

kubeadm init \
  --apiserver-advertise-address= \
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.17.0 \


W1221 17:44:19.880281    2865 version.go:101] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1221 17:44:19.880405    2865 version.go:102] falling back to the local client version: v1.17.0
W1221 17:44:19.880539    2865 validation.go:28] Cannot validate kube-proxy config - no validator is available
W1221 17:44:19.880546    2865 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [ubuntu kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs []
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [ubuntu localhost] and IPs [ ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [ubuntu localhost] and IPs [ ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W1221 17:50:12.262505    2865 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W1221 17:50:12.268198    2865 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 17.504683 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node ubuntu as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node ubuntu as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 1rpp8b.axfud1xrsvx4q8nw
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join --token 1rpp8b.axfud1xrsvx4q8nw \
    --discovery-token-ca-cert-hash sha256:6bf952d45bbdc121fa90583eac33f11f0a3f4b491f29996a56fc289363843e3c 

Version 2: pull the version according to the previous script:

kubeadm init --pod-network-cidr=

Note: problems may be encountered after successful initialization:

1, [ERROR Port-10251]: Port 10250 is in use

terms of settlement:

Restart kubeadm: [root @ k8s master ~], and kubeadm reset

If the port is still occupied after restart, check the process of occupying the port: netstat - tunlp | grep 10250

Force to kill the process occupying the port: sudo fuser -k -n tcp 10250


After the deployment is successful, copy the admin.conf file to the corresponding directory of the current user according to the prompt. The admin.conf file will be used later (it needs to be copied to the node node).

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

During initialization, if it does not exist, the image will be downloaded automatically. After initialization, the image is as follows:

# docker images
REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
registry.aliyuncs.com/google_containers/kube-proxy                v1.17.4             7d54289267dc        1 days ago         116MB
registry.aliyuncs.com/google_containers/kube-apiserver            v1.17.4             0cae8d5cc64c        1 days ago         171MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.17.4            5eb3b7486872        1 days ago         161MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.17.4             78c190f736b1        1 days ago         94.4MB
registry.aliyuncs.com/google_containers/coredns                   1.6.5               70f311871ae1        6 weeks ago         41.6MB
registry.aliyuncs.com/google_containers/etcd                      3.4.3-0             303ce5db0e90        8 weeks ago         288MB
registry.aliyuncs.com/google_containers/pause                     3.1                 da86e6ba6ca1        2 years ago         742kB

The pod status is as follows:

# kubectl get pods -n kube-system
NAME                             READY   STATUS    RESTARTS   AGE
coredns-9d85f5447-67qtv          0/1     Pending   0          3h26m
coredns-9d85f5447-cg87c          0/1     Pending   0          3h26m
etcd-ubuntu                      1/1     Running   0          3h27m
kube-apiserver-ubuntu            1/1     Running   0          3h27m
kube-controller-manager-ubuntu   1/1     Running   0          3h27m
kube-proxy-chqbq                 1/1     Running   0          3h26m
kube-scheduler-ubuntu            1/1     Running   0          3h27m

Except that coredns status is Pending, all other pod s are running. This is because the network plug-in is not deployed. This paper chooses flannel.

Execute the following command to deploy flannel:

# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Deploy using the kube-flannel.yml file of the flannel repository. Refer to this document for details.
If you can't access it, you can manually download the https://github.com/cores/flannel/blob/master/documentation/kube-flannel.yml file to the current directory, and then execute the kubectl apply -f kube-flannel.yml command.

# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created

3, Node node

k8s is deployed by master and node nodes. This section is node node.

3.1 preconditions
Operate on a node node.
1. Install kubeadm, see above.
2. Download the flannel image, as mentioned above (if it is not downloaded in advance, it will be downloaded automatically when joining the cluster).
3. Copy the / etc/kubernetes/admin.conf file of the host to the / etc/kubernetes / directory of the node node. (Note: use scp command in master node, kubernetes does not exist)

3.2 join the cluster
At this time, the k8s service has not been started. Execute the following command to join the node:

kubeadm join --token 1rpp8b.axfud1xrsvx4q8nw \
    --discovery-token-ca-cert-hash sha256:6bf952d45bbdc121fa90583eac33f11f0a3f4b491f29996a56fc289363843e3c 

The prompt message is as follows:

[preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

During the process of joining the cluster, the necessary k8s image will be downloaded. Note that the master computer has been designated as the source of the Ali source, so it is also the source on the node node.

REPOSITORY                                           TAG                 IMAGE ID            CREATED             SIZE
registry.aliyuncs.com/google_containers/kube-proxy   v1.17.0             7d54289267dc        2 weeks ago         116MB
registry.aliyuncs.com/google_containers/coredns      1.6.5               70f311871ae1        7 weeks ago         41.6MB
quay.io/coreos/flannel                               v0.11.0-amd64       ff281650a721        11 months ago       52.6MB
registry.aliyuncs.com/google_containers/pause        3.1                 da86e6ba6ca1        2 years ago         742kB

After joining successfully, the following related services are running in this node:

# ps aux | grep kube
root       3269  1.6  4.2 754668 86784 ?        Ssl  Dec20  18:34 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.1
root       3632  0.1  1.1 140104 22412 ?        Ssl  Dec20   2:14 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=node
root       4385  0.0  1.6 407356 33704 ?        Ssl  Dec20   0:51 /opt/bin/flanneld --ip-masq --kube-subnet-mgr
root     121292  0.0  0.0  14228  1032 pts/0    S+   00:33   0:00 grep --color=auto kube

It mainly includes kubelet, Kube proxy, flanneld, etc.

The docker container list is as follows:

# docker ps
CONTAINER ID        IMAGE                                                COMMAND                  CREATED             STATUS              PORTS               NAMES
2fde9bb78fd7        ff281650a721                                         "/opt/bin/flanneld -..."   7 minutes ago       Up 7 minutes                            k8s_kube-flannel_kube-flannel-ds-amd64-28p6z_kube-system_f40a2875-70eb-468b-827d-fcb59be3416b_1
aa7ca3d5825e        registry.aliyuncs.com/google_containers/kube-proxy   "/usr/local/bin/kube..."   8 minutes ago       Up 8 minutes                            k8s_kube-proxy_kube-proxy-n6xv5_kube-system_3df8b7ae-e5b8-4256-9857-35bd24f7e025_0
ac61ed8d7295        registry.aliyuncs.com/google_containers/pause:3.1    "/pause"                 8 minutes ago       Up 8 minutes                            k8s_POD_kube-flannel-ds-amd64-28p6z_kube-system_f40a2875-70eb-468b-827d-fcb59be3416b_0
423f9e42c082        registry.aliyuncs.com/google_containers/pause:3.1    "/pause"                 8 minutes ago       Up 8 minutes                            k8s_POD_kube-proxy-n6xv5_kube-system_3df8b7ae-e5b8-4256-9857-35bd24f7e025_0

To view the flannel network information:

# cat /run/flannel/subnet.env 

To view local IP information:

# ifconfig 

Five, validation

Execute in the master node:

# kubectl get nodes       
node     Ready    <none>   17m     v1.17.0
ubuntu   Ready    master   5h11m   v1.17.0

You can see that both machines are Ready. The node machine changes from NotReady to Ready, which takes about 10 seconds.

Simply test pod with busybox image. Execute in the master node:

# kubectl run -i --tty busybox --image=latelee/busybox --restart=Never -- sh

Wait a moment to enter the busybox command line:

 # uname -a
Linux busybox 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 GNU/Linux

From another command line, check the pod running status:

# kubectl get pod -o wide 
busybox   1/1     Running   0          74s   node   <none>           <none>

You can see that the pod is in Running status and Running on the node.
View on the node node:

# docker ps | grep busybox
ba5d1a480294        latelee/busybox                                      "sh"                     2 minutes ago       Up 2 minutes                            k8s_busybox_busybox_default_20d757f7-8ea7-4e51-93fc-514029065a59_0
8c643171ac09        registry.aliyuncs.com/google_containers/pause:3.1    "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_busybox_default_20d757f7-8ea7-4e51-93fc-514029065a59_0

At this time, when the master node exits from busybox, the pod still exists, but it is not READY, and the node host does not have a busybox container running.

After verification, k8s deployment succeeded.


Reference documents: https://blog.csdn.net/subfate/article/details/103774072

Published 31 original articles, won praise 14, visited 20000+
Private letter follow

Posted by steamPunk on Tue, 17 Mar 2020 00:37:39 -0700