k8s using Kube router to build highly available and extensible ingress

Keywords: Web Server curl Nginx network DNS

brief introduction

Use Kube router to realize the function of inress of k8s cluster, which is highly available and easy to expand

Environmental description

This experiment is based on the k8s cluster which has been installed and configured. For k8s installation, please refer to other blog articles. lab4, as a router, forwards lab5's request

Experimental framework

lab1: master
lab2: node
lab3: node
lab4: router
lab5: client
 Copy code


# In this experiment, the cluster was recreated. The cluster environment of other network plug-ins tested before was not successful # It may be due to environmental interference, so attention shall be paid during the experiment # Create Kube router directory to download related files
mkdir kube-router && cd kube-router
rm -f generic-kuberouter-all-features-dsr.yaml
wget https://raw.githubusercontent.com/mgxian/kube-router/master/generic-kuberouter-all-features-dsr.yaml

# Enable pod All functions of network communication, network isolation strategy and service agent # CLUSTERCIDR kube-controller-manager startup parameter --cluster-cidr Value # Apiserver Kube apiserver startup parameter -- advertize address value
sed -i "s;%APISERVER%;$APISERVER;g" generic-kuberouter-all-features-dsr.yaml
sed -i "s;%CLUSTERCIDR%;$CLUSTERCIDR;g" generic-kuberouter-all-features-dsr.yaml

# Modify configuration
 - name: kube-router
 image: cloudnativelabs/kube-router
 imagePullPolicy: Always
 - --peer-router-ips=
 - --peer-router-asns=64513
 - --cluster-asn=64512
 - --advertise-external-ip=true

# deploy
kubectl apply -f generic-kuberouter-all-features-dsr.yaml

# Delete Kube proxy
kubectl -n kube-system delete ds kube-proxy

# Execute on each node # For binary installation, use the following command
systemctl stop kube-proxy

# Execute on each node # Clean up rules left by Kube proxy
docker run --privileged --net=host registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:v1.10.2 kube-proxy --cleanup

# See
kubectl get pods -n kube-system
kubectl get svc -n kube-system
//Copy code


# Please install and configure before testing kube-dns perhaps coredns # Start deployment for testing
kubectl run nginx --replicas=2 --image=nginx:alpine --port=80
kubectl expose deployment nginx --type=NodePort --name=example-service-nodeport
kubectl expose deployment nginx --name=example-service

# See
kubectl get pods -o wide
kubectl get svc -o wide

# dns and access testing
kubectl run curl --image=radial/busyboxplus:curl -i --tty
nslookup kubernetes
nslookup example-service
curl example-service
//Copy code

Configure quagga in lab4

# install
yum install -y quagga

# To configure
cat >/etc/quagga/bgpd.conf<<EOF
! -*- bgp -*-
! BGPd sample configuratin file
! $Id: bgpd.conf.sample,v 1.1 2002/12/13 20:15:29 paul Exp $
hostname lab4
password password
router bgp 64513
 bgp router-id
 maximum-paths 4
 neighbor remote-as 64512
 neighbor remote-as 64512
 neighbor remote-as 64512
log stdout

# start-up
systemctl start bgpd
systemctl status bgpd
systemctl enable bgpd

# View route information
ip route
//Copy code

Test in lab4

# Modify example service to configure external ip on lab1
kubectl edit svc example-service

# stay lab1 View up svc information # You can see that example service has external ip
kubectl get svc

# See lab4 Route # You can see that there are related routes
ip route

# Accessing tests on lab4
//Copy code

Test in lab5

#Add route in lab5
ip route add via
ip route

#Access test in lab5

#View ipvs in lab1
ipvsadm -L -n
 Copy code

Using DSR

#The DSR experiment failed. The experimental environment is that vagrant cooperates with virtualbox to set example service in lab1 to use DSR mode. The response of the service is sent directly to the client without lvs transfer
 Kubectl annotate SVC example service "kubectl router. IO / service. DSR = Tunnel" ා check ipvs in lab1 ා you can see Tunnel forwarding type
ipvsadm -L -n

#Access test in lab5

#Analysis of node packet capturing in cluster
tcpdump -i kube-bridge proto 4
 Copy code


Cleaning up
kubectl delete svc example-service example-service-nodeport
kubectl delete deploy nginx curl
 Copy code

Reference document

  • https://cloudnativelabs.github.io/post/2017-11-01-kube-high-available-ingress/
  • https://github.com/cloudnativelabs/kube-router/blob/master/docs/generic.md

This article turns to gold digging- k8s using Kube router to build highly available and extensible ingress

Posted by eirikol on Tue, 03 Dec 2019 05:27:35 -0800