CoreDNS implements automatic discovery of services, so how do we expose our services?
The first is a nodePort-type service: however, it cannot use the ipvs model, only the iptables model
The second option is ingress: note that Ingres resources can only schedule seven-tier network resources, specifically http/https
ingress is one of the standard resource types of the k8s API as well as a core resource. It is actually a set of domain name-based URL paths, which forwards user requests to the established rules of serivce resources, forwards external traffic to the internal, thereby exposing services
The software commonly used to implement ingress is:
Haproxy
ingress-nginx
fraefik
Here we use fraefik as our ingress controller:
Prepare the fraefik image:
[root@hdss7-200 ~]# docker pull traefik:v1.7.2-alpine v1.7.2-alpine: Pulling from library/traefik 4fe2ade4980c: Pull complete 8d9593d002f4: Pull complete 5d09ab10efbd: Pull complete 37b796c58adc: Pull complete Digest: sha256:cf30141936f73599e1a46355592d08c88d74bd291f05104fe11a8bcce447c044 Status: Downloaded newer image for traefik:v1.7.2-alpine docker.io/library/traefik:v1.7.2-alpine [root@hdss7-200 ~]# [root@hdss7-200 ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE goharbor/chartmuseum-photon v0.9.0-v1.8.3 ec654bcf3624 6 months ago 131MB goharbor/harbor-migrator v1.8.3 6f945bb96ea3 6 months ago 362MB goharbor/redis-photon v1.8.3 cda8fa1932ec 6 months ago 109MB goharbor/clair-photon v2.0.8-v1.8.3 5630fa937f6d 6 months ago 165MB goharbor/notary-server-photon v0.6.1-v1.8.3 e0a54affd0c8 6 months ago 136MB goharbor/notary-signer-photon v0.6.1-v1.8.3 72708cdfb905 6 months ago 133MB goharbor/harbor-registryctl v1.8.3 9dc783842a19 6 months ago 97.2MB goharbor/registry-photon v2.7.1-patch-2819-v1.8.3 a05e085842f5 6 months ago 82.3MB goharbor/nginx-photon v1.8.3 3a016e0dc7de 6 months ago 37MB goharbor/harbor-log v1.8.3 b92621c47043 6 months ago 82.6MB goharbor/harbor-jobservice v1.8.3 53bc2359083f 6 months ago 120MB goharbor/harbor-core v1.8.3 a3ccc3897bc0 6 months ago 136MB goharbor/harbor-portal v1.8.3 514f2fb70e90 6 months ago 43.9MB goharbor/harbor-db v1.8.3 d1b8adbed58f 6 months ago 147MB goharbor/prepare v1.8.3 a37e777b7fe7 6 months ago 147MB coredns/coredns 1.6.1 c0f6e815079e 7 months ago 42.2MB harbor.od.com/public/coredns v1.6.1 c0f6e815079e 7 months ago 42.2MB traefik v1.7.2-alpine add5fac61ae5 18 months ago 72.4MB nginx 1.7.9 84581e99d807 5 years ago 91.7MB harbor.od.com/public/nginx v1.7.9 84581e99d807 5 years ago 91.7MB kubernetes/pause latest f9d5de079539 5 years ago 240kB harbor.od.com/public/pause latest f9d5de079539 5 years ago 240kB [root@hdss7-200 ~]# docker tag add5fac61ae5 harbor.od.com/public/traefik:v1.7.2 [root@hdss7-200 ~]# docker push harbor.od.com/public/traefik:v1.7.2 The push refers to repository [harbor.od.com/public/traefik] a02beb48577f: Pushed ca22117205f4: Pushed 3563c211d861: Pushed df64d3292fd6: Pushed v1.7.2: digest: sha256:6115155b261707b642341b065cd3fac2b546559ba035d0262650b3b3bbdd10ea size: 1157
Prepare resource allocation checklist:
# cat rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: traefik-ingress-controller namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: traefik-ingress-controller rules: - apiGroups: - "" resources: - services - endpoints - secrets verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses verbs: - get - list - watch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: traefik-ingress-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: traefik-ingress-controller subjects: - kind: ServiceAccount name: traefik-ingress-controller namespace: kube-system # cat ds.yaml apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: traefik-ingress namespace: kube-system labels: k8s-app: traefik-ingress spec: template: metadata: labels: k8s-app: traefik-ingress name: traefik-ingress spec: serviceAccountName: traefik-ingress-controller terminationGracePeriodSeconds: 60 containers: - image: harbor.od.com/public/traefik:v1.7.2 name: traefik-ingress ports: - name: controller containerPort: 80 hostPort: 81 - name: admin-web containerPort: 8080 securityContext: capabilities: drop: - ALL add: - NET_BIND_SERVICE args: - --api - --kubernetes - --logLevel=INFO - --insecureskipverify=true - --kubernetes.endpoint=https://10.4.7.10:7443 - --accesslog - --accesslog.filepath=/var/log/traefik_access.log - --traefiklog - --traefiklog.filepath=/var/log/traefik.log - --metrics.prometheus # cat ingress.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: traefik-web-ui namespace: kube-system annotations: kubernetes.io/ingress.class: traefik spec: rules: - host: traefik.od.com http: paths: - path: / backend: serviceName: traefik-ingress-service servicePort: 8080 # cat svc.yaml kind: Service apiVersion: v1 metadata: name: traefik-ingress-service namespace: kube-system spec: selector: k8s-app: traefik-ingress ports: - protocol: TCP port: 80 name: controller - protocol: TCP port: 8080 name: admin-web
Use the declarative resource management method to apply our declarative resource allocation list:
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/traefik/rbac.yaml serviceaccount/traefik-ingress-controller created clusterrole.rbac.authorization.k8s.io/traefik-ingress-controller created clusterrolebinding.rbac.authorization.k8s.io/traefik-ingress-controller created [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/traefik/ds.yaml daemonset.extensions/traefik-ingress created [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/traefik/svc.yaml service/traefik-ingress-service created [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/traefik/ingress.yaml ingress.extensions/traefik-web-ui created
Check if the pod status is up:
~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-6b6c4f9648-j7cv9 1/1 Running 0 82m traefik-ingress-4pdm5 0/1 ContainerCreating 0 4s traefik-ingress-rgcqp 0/1 ContainerCreating 0 29s # kubectl describe pod -n kube-system traefik-ingress-4pdm5 Warning FailedCreatePodSandBox 7s kubelet, hdss7-22.host.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "traefik-ingress-4pdm5": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_traefik-ingress-4pdm5_kube-system_8d6fb147-074c-46b3-b5a0-7cff176671ec_8 (a840cdb6e9da00aefc7ce6d233a373acf4ecef3ee06890fb647208069ed59f25): (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.7.22.3 --dport 80 -j ACCEPT: iptables: No chain/target/match by that name.
After restarting the docker process, you've found that it's OK
[root@hdss7-21 ~]# systemctl restart docker [root@hdss7-22 ~]# systemctl restart docker [root@hdss7-21 ~]# kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES coredns-6b6c4f9648-j7cv9 1/1 Running 0 85m 172.7.21.4 hdss7-21.host.com <none> <none> traefik-ingress-4pdm5 1/1 Running 0 2m59s 172.7.22.3 hdss7-22.host.com <none> <none> traefik-ingress-rgcqp 1/1 Running 0 3m24s 172.7.21.5 hdss7-21.host.com <none> <none>
Configure fraefik domain name resolution:
[root@hdss7-11 named]# cat od.com.zone $ORIGIN od.com. $TTL 600; 10 minutes @ IN SOAdns.od.com. dnsadmin.od.com. ( 2019111004 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.od.com. $TTL 60; 1 minute dns A 10.4.7.11 harbor A 10.4.7.200 k8s-yaml A 10.4.7.200 fraefik A 10.4.7.11 [root@hdss7-11 named]# systemctl restart named [root@hdss7-11 named]# dig @10.4.7.11 fraefik.od.com +short 10.4.7.11
Then on the ingress entry host, we add the following configuration for nginx, explaining: We make a generic match of business domains, and then throw all the rules to port 81 on the ingress node. This way, the configuration of nginx is like adding our rules to the resource configuration list if ingress does not have machine-offline operations at allThen, give the routing rules of the business entirely to the resource configuration list
[root@hdss7-200 conf.d]# cat od.com.conf
upstream default_backend_traefik {
server 10.4.7.21:81 max_fails=3 fail_timeout=10s;
server 10.4.7.22:81 max_fails=3 fail_timeout=10s;
}
server {
server_name *.od.com;
location / {
proxy_pass http://default_backend_traefik;
proxy_set_header Host $http_host;
proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
}
}
[root@hdss7-200 conf.d]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@hdss7-200 conf.d]# nginx -s reload