k8s Service Exposure Plugin-fraefik

Keywords: Nginx Docker iptables Kubernetes

CoreDNS implements automatic discovery of services, so how do we expose our services?

The first is a nodePort-type service: however, it cannot use the ipvs model, only the iptables model

The second option is ingress: note that Ingres resources can only schedule seven-tier network resources, specifically http/https


ingress is one of the standard resource types of the k8s API as well as a core resource. It is actually a set of domain name-based URL paths, which forwards user requests to the established rules of serivce resources, forwards external traffic to the internal, thereby exposing services

The software commonly used to implement ingress is:

Haproxy

ingress-nginx

fraefik

Here we use fraefik as our ingress controller:

Prepare the fraefik image:

[root@hdss7-200 ~]# docker pull traefik:v1.7.2-alpine
v1.7.2-alpine: Pulling from library/traefik
4fe2ade4980c: Pull complete
8d9593d002f4: Pull complete
5d09ab10efbd: Pull complete
37b796c58adc: Pull complete
Digest: sha256:cf30141936f73599e1a46355592d08c88d74bd291f05104fe11a8bcce447c044
Status: Downloaded newer image for traefik:v1.7.2-alpine
docker.io/library/traefik:v1.7.2-alpine
[root@hdss7-200 ~]#
[root@hdss7-200 ~]# docker images
REPOSITORY                      TAG                        IMAGE ID            CREATED             SIZE
goharbor/chartmuseum-photon     v0.9.0-v1.8.3              ec654bcf3624        6 months ago        131MB
goharbor/harbor-migrator        v1.8.3                     6f945bb96ea3        6 months ago        362MB
goharbor/redis-photon           v1.8.3                     cda8fa1932ec        6 months ago        109MB
goharbor/clair-photon           v2.0.8-v1.8.3              5630fa937f6d        6 months ago        165MB
goharbor/notary-server-photon   v0.6.1-v1.8.3              e0a54affd0c8        6 months ago        136MB
goharbor/notary-signer-photon   v0.6.1-v1.8.3              72708cdfb905        6 months ago        133MB
goharbor/harbor-registryctl     v1.8.3                     9dc783842a19        6 months ago        97.2MB
goharbor/registry-photon        v2.7.1-patch-2819-v1.8.3   a05e085842f5        6 months ago        82.3MB
goharbor/nginx-photon           v1.8.3                     3a016e0dc7de        6 months ago        37MB
goharbor/harbor-log             v1.8.3                     b92621c47043        6 months ago        82.6MB
goharbor/harbor-jobservice      v1.8.3                     53bc2359083f        6 months ago        120MB
goharbor/harbor-core            v1.8.3                     a3ccc3897bc0        6 months ago        136MB
goharbor/harbor-portal          v1.8.3                     514f2fb70e90        6 months ago        43.9MB
goharbor/harbor-db              v1.8.3                     d1b8adbed58f        6 months ago        147MB
goharbor/prepare                v1.8.3                     a37e777b7fe7        6 months ago        147MB
coredns/coredns                 1.6.1                      c0f6e815079e        7 months ago        42.2MB
harbor.od.com/public/coredns    v1.6.1                     c0f6e815079e        7 months ago        42.2MB
traefik                         v1.7.2-alpine              add5fac61ae5        18 months ago       72.4MB
nginx                           1.7.9                      84581e99d807        5 years ago         91.7MB
harbor.od.com/public/nginx      v1.7.9                     84581e99d807        5 years ago         91.7MB
kubernetes/pause                latest                     f9d5de079539        5 years ago         240kB
harbor.od.com/public/pause      latest                     f9d5de079539        5 years ago         240kB
[root@hdss7-200 ~]# docker tag add5fac61ae5 harbor.od.com/public/traefik:v1.7.2
[root@hdss7-200 ~]# docker push harbor.od.com/public/traefik:v1.7.2
The push refers to repository [harbor.od.com/public/traefik]
a02beb48577f: Pushed
ca22117205f4: Pushed
3563c211d861: Pushed
df64d3292fd6: Pushed
v1.7.2: digest: sha256:6115155b261707b642341b065cd3fac2b546559ba035d0262650b3b3bbdd10ea size: 1157

Prepare resource allocation checklist:

# cat rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system
# cat ds.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: traefik-ingress
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress
spec:
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress
        name: traefik-ingress
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: harbor.od.com/public/traefik:v1.7.2
        name: traefik-ingress
        ports:
        - name: controller
          containerPort: 80
          hostPort: 81
        - name: admin-web
          containerPort: 8080
        securityContext:
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        args:
        - --api
        - --kubernetes
        - --logLevel=INFO
        - --insecureskipverify=true
        - --kubernetes.endpoint=https://10.4.7.10:7443
        - --accesslog
        - --accesslog.filepath=/var/log/traefik_access.log
        - --traefiklog
        - --traefiklog.filepath=/var/log/traefik.log
        - --metrics.prometheus
# cat ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: traefik.od.com
    http:
      paths:
      - path: /
        backend:
          serviceName: traefik-ingress-service
          servicePort: 8080
# cat svc.yaml
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress
  ports:
    - protocol: TCP
      port: 80
      name: controller
    - protocol: TCP
      port: 8080
      name: admin-web

Use the declarative resource management method to apply our declarative resource allocation list:

[root@hdss7-21 ~]# kubectl apply -f  http://k8s-yaml.od.com/traefik/rbac.yaml
serviceaccount/traefik-ingress-controller created
clusterrole.rbac.authorization.k8s.io/traefik-ingress-controller created
clusterrolebinding.rbac.authorization.k8s.io/traefik-ingress-controller created
[root@hdss7-21 ~]# kubectl apply -f  http://k8s-yaml.od.com/traefik/ds.yaml
daemonset.extensions/traefik-ingress created
[root@hdss7-21 ~]# kubectl apply -f  http://k8s-yaml.od.com/traefik/svc.yaml
service/traefik-ingress-service created
[root@hdss7-21 ~]# kubectl apply -f  http://k8s-yaml.od.com/traefik/ingress.yaml
ingress.extensions/traefik-web-ui created

Check if the pod status is up:

~]# kubectl get pod -n kube-system
NAME                       READY   STATUS              RESTARTS   AGE
coredns-6b6c4f9648-j7cv9   1/1     Running             0          82m
traefik-ingress-4pdm5      0/1     ContainerCreating   0          4s
traefik-ingress-rgcqp      0/1     ContainerCreating   0          29s
# kubectl describe pod -n kube-system traefik-ingress-4pdm5
  Warning  FailedCreatePodSandBox  7s  kubelet, hdss7-22.host.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "traefik-ingress-4pdm5": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_traefik-ingress-4pdm5_kube-system_8d6fb147-074c-46b3-b5a0-7cff176671ec_8 (a840cdb6e9da00aefc7ce6d233a373acf4ecef3ee06890fb647208069ed59f25):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.7.22.3 --dport 80 -j ACCEPT: iptables: No chain/target/match by that name.

After restarting the docker process, you've found that it's OK

[root@hdss7-21 ~]# systemctl restart docker
[root@hdss7-22 ~]# systemctl restart docker
[root@hdss7-21 ~]# kubectl get pod -n kube-system -o wide
NAME                       READY   STATUS    RESTARTS   AGE     IP           NODE                NOMINATED NODE   READINESS GATES
coredns-6b6c4f9648-j7cv9   1/1     Running   0          85m     172.7.21.4   hdss7-21.host.com   <none>           <none>
traefik-ingress-4pdm5      1/1     Running   0          2m59s   172.7.22.3   hdss7-22.host.com   <none>           <none>
traefik-ingress-rgcqp      1/1     Running   0          3m24s   172.7.21.5   hdss7-21.host.com   <none>           <none>

Configure fraefik domain name resolution:

[root@hdss7-11 named]# cat od.com.zone
$ORIGIN od.com.
$TTL 600; 10 minutes
@   IN SOAdns.od.com. dnsadmin.od.com. (
2019111004 ; serial
10800      ; refresh (3 hours)
900        ; retry (15 minutes)
604800     ; expire (1 week)
86400      ; minimum (1 day)
)
NS   dns.od.com.
$TTL 60; 1 minute
dns                  A    10.4.7.11
harbor               A    10.4.7.200
k8s-yaml             A    10.4.7.200
fraefik              A    10.4.7.11
[root@hdss7-11 named]# systemctl restart named
[root@hdss7-11 named]# dig @10.4.7.11 fraefik.od.com +short
10.4.7.11

Then on the ingress entry host, we add the following configuration for nginx, explaining: We make a generic match of business domains, and then throw all the rules to port 81 on the ingress node. This way, the configuration of nginx is like adding our rules to the resource configuration list if ingress does not have machine-offline operations at allThen, give the routing rules of the business entirely to the resource configuration list

[root@hdss7-200 conf.d]# cat od.com.conf
upstream default_backend_traefik {
    server 10.4.7.21:81    max_fails=3 fail_timeout=10s;
    server 10.4.7.22:81    max_fails=3 fail_timeout=10s;
}
server {
    server_name *.od.com;
    location / {
        proxy_pass http://default_backend_traefik;
        proxy_set_header Host            $http_host;
        proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
    }
}
[root@hdss7-200 conf.d]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@hdss7-200 conf.d]# nginx -s reload


Posted by ChrisF79 on Fri, 27 Mar 2020 20:14:39 -0700