1, Overview
Kubernetes exposes services in three ways: loadlander Service, NodePort Service and progress. The official website defines ingress as a collection of rules for managing external services to services in the cluster. Generally speaking, it defines rules to allow requests to enter the cluster to be forwarded to the corresponding services in the cluster, never realizing Service leakage. Ingress can configure the services in the cluster into URL s that can be accessed by the external network, traffic load balancing, terminating SSL, providing virtual hosts based on domain name access, etc.
1)LoadBlancer Service
Loadbalancer service is a component of Kubernetes combined with cloud platforms, such as foreign GCE, AWS, domestic alicloud, etc. it is used to apply to the underlying cloud platform to create a load balancer for implementation. It has limitations and is more convenient for clusters using cloud platforms.
2)NodePort Service
NodePort Service realizes Service leakage by exposing ports on nodes, and then mapping ports to a specific Service. It is more intuitive and convenient. However, for clusters, with the continuous increase of services, more and more ports are required, which is prone to end port conflict and difficult to manage. Of course, it is quite good for small-scale cluster services.
3)Ingress
Ingress uses the open source reverse proxy load balancer to implement external disclosure services, such as nginx, Apache, Haproxy, etc. Nginx Ingress generally consists of three components:
- ingress is a resource object of kubernetes, which is used to write definition rules.
- The reverse proxy load balancer usually runs in the Port mode of Service, receives and forwards according to the rules defined by ingress, usually nginx, haproxy, traefik, etc. nginx is used in this paper.
- The ingress controller monitors the apiserver, obtains changes such as service addition and deletion, and dynamically updates it to the reverse proxy load balancer in combination with the ingress rules, and reloads the configuration to make it effective.
With the organic coordination of the above three, the leakage of Kubernetes cluster services can be completed.
2, Introduction to ingress nginx
1) Ingress nginx composition
- Ingress nginx controller: dynamically change the configuration file of nginx service according to the ingress rules written by the user (the yaml file of ingress created), and reload it to make it effective (it is automated and implemented through lua script)
- Ingress resource object: abstract the configuration of Nginx into an ingress object. Each time you add a new Service resource object, you only need to write a new ingress rule yaml file (or modify the existing ingress rule yaml file)
2) Ingress nginx workflow
The implementation of Ingress is divided into two parts: Ingress Controller and Ingress
Up to now, Nginx ingress has been able to complete the 7 / 4-layer proxy function (the 4-layer proxy is based on ConfigMap, and I feel there is still room for improvement); The 7-layer reverse proxy mode of Nginx can be simply represented by the following figure:
- Nginx provides reverse proxy for the back-end running services (Service1 and Service2), and configures the corresponding relationship between the domain name and the back-end service Endpoints in the configuration file.
- The client maps the domain name to the Nginx proxy server by using the DNS service or directly configuring the local hosts file.
- When the client accesses service1.com, the browser will send the request containing the domain name to the nginx server. The nginx server selects the corresponding Service according to the transmitted domain name. Here, select the back-end Service of Service 1, and then select a container in Service1 to receive the request from the client and respond according to a certain load balancing policy.
The process is very simple. nginx seems to be a "router" that forwards requests according to the domain name, which is the overall workflow of the layer 7 agent!
3) Working principle
We have a general idea of what the nginx reverse proxy does. In k8s system, the change of back-end services is very frequent, and it is almost impossible to update nginx configuration files only by manual. Therefore, nginx ingress came into being. Nginx ingress automatically updates the nginx configuration file by monitoring k8s the change of resource status. This paper will analyze its working principle.
- The nginx ingress module mainly includes three subjects at runtime: NginxController, Store and SyncQueue.
- Store is mainly responsible for collecting runtime information from kubernetes APIServer, sensing the changes of various resources (such as inress, service, etc.), and timely writing update event messages (events) to a ring pipe.
- The syncQueue collaboration regularly scans the syncQueue queue. If a task is found, it executes the update operation, that is, pull the latest running data with the help of the Store, and then generate a new nginx configuration according to certain rules (some updates must be reload, write the new configuration locally, and execute reload), and then execute the dynamic update operation, that is, construct post data, Send a post request to the local Nginx Lua service module to update the configuration.
- As an intermediate contact, NginxController listens to updateChannel. Once a configuration update event is received, it writes an update request to the synchronization queue syncQueue.
3, Install ingress nginx
1) Official introduction
ingress Official website
ingress Warehouse address
Inress nginx v1.0 latest version v1.0
For Kubernetes version v1.19 + (including v1.19)
Kubernetes-v1.22 + requires ingress nginx > = 1.0 because networking.k8s.io/v1beta has been removed
2) Deploy ingress nginx directly
Direct deployment is relatively simple. Just pull the girhub file directly. If there is no response for a long time, you can terminate the task and pull it again.
Pull image
$ wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.0/deploy/static/provider/baremetal/deploy.yaml
If the above address fails to download, you can use my address to download. If you use the official website, you may need to change the image address, because the image downloaded from the official website may fail. Here I use the following yaml file to create the ingress controller
Link: https://pan.baidu.com/s/1fbqcy2bJMtmziJoXTbdK2g
Extraction code: 8888
# Modify mirror address $ sed -i 's@k8s.gcr.io/ingress-nginx/controller:v1.0.0\(.*\)@willdockerhub/ingress-nginx-controller:v1.0.0@' deploy.yaml $ sed -i 's@k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0\(.*\)$@hzde0128/kube-webhook-certgen:v1.0@' deploy.yaml $ kubectl apply -f deploy.yaml
implement
$ kubectl apply -f ingress-nginx.yaml
Check installation
The Completed status is normal and can be ignored.
$ kubectl get pod -n ingress-nginx $ kubectl get svc -n ingress-nginx $ kubectl get svc -n ingress-nginx
4, Creating nginx apps
1) Create directory
$ mkdir -p /opt/ingress/nginx-test $ cd /opt/ingress/nginx-test
2) Create nginx-Deployment-Service.yaml file as follows:
$ cat << EOF > nginx-Deployment-Service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
minReadySeconds: 1
progressDeadlineSeconds: 60
revisionHistoryLimit: 2
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.17.1
imagePullPolicy: Always
ports:
- containerPort: 80
resources:
requests:
memory: "1Gi"
cpu: "80m"
limits:
memory: "1Gi"
cpu: "80m"
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
labels:
app: nginx
spec:
selector:
app: nginx
ports:
- name: nginx-port
protocol: TCP
port: 80
targetPort: 80
type: ClusterIP
EOF
3) Deploy nginx applications
$ kubectl apply -f nginx-Deployment-Service.yaml $ kubectl get svc -o wide|grep nginx-service $ kubectl get pod -o wide|grep nginx-deployment-*
4) Create the ingress yaml file as follows:
cat << EOF > nginx-Ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: ingress.nginx.com
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: nginx-service
port:
number: 80
EOF
establish
$ kubectl apply -f nginx-Ingress.yaml $ kubectl get ingress|grep nginx
5) Append the IP address of the ingress node at the end of the hosts file
$ kubectl get ingress|grep nginx $ cat /etc/hosts $ kubectl get svc -n ingress-nginx
6) Verify
curl -I http://ingress.nginx.com:31473
5, Deploy ingress nginx using the host network
Every time you deploy ingress nginx, a nodePort is randomly assigned. When you use ingress nginx to access, you also need to access it in the form of domain name: port. How can you directly use the domain name to access it? Another installation method is described below.
1) Create directory
$ mkdir /opt/ingress/nginx-test/hostNetwork $ cd /opt/ingress/nginx-test/hostNetwork
2) Copy a copy of deploy.yaml
$ cp /opt/ingress/deploy.yaml /opt/ingress/nginx-test/hostNetwork/
3) Optimize ingress nginx
1. Using hostNetwork
By default, ingress nginx randomly provides nodeport port. Open hostNetwork and enable ports 80 and 443.
Modify the spec under Deployment
The parameters are as follows:
hostNetwork: true # NEW
2. Modify load balancing issues
Change the kind: Deployment mode to the kind: daemon mode, so that each node has a copy of the ingress nginx controller pod.
The parameters are as follows:
#kind: Deployment # comment
kind: DaemonSet # added
3. Modify ingressClass problem
If you don't care about the ingress class or many ingress objects without an ingress class configuration,
Add the parameter ingress controller -- watch ingress without class = true.
4. Deployment check ingress
$ kubectl apply -f deploy.yaml
It is found that all node nodes except the master node have a replica.
View nodes on other nodes
$ netstat -pntl |grep :443 $ netstat -pntl |grep :80
5. Verify
# Direct access, not through nodePort $ curl -I http://ingress.nginx.com # Of course, it can also be accessed through nodePort $ curl -I http://ingress.nginx.com:31827
6, HTTPS based ingress nginx
1) Create CA certificate
$ mkdir -p /opt/ingress/tls $ cd /opt/ingress/tls # Generate ca certificate. Parameters can be customized according to requirements $ openssl genrsa -out tls.key 2048 $ openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Shanghai/L=Shanghai/O=DevOps/CN=ingress.nginx.com
PS: CN must be the same as the domain name accessed
After creation, certificate key (tls.crt) and certificate key (tls.key) files will be generated in the current directory:
2) Introduction to secret
Secret is used to store and manage some sensitive data, such as passwords, token s, keys and other sensitive information. It stores the encrypted data that the Pod wants to access in Etcd. Then, users can access the information stored in these secrets by mounting volumes in the Pod container or environment variables.
Secret can be divided into four types:
- kubernetes.io/dockerconfigjson: used to store the authentication information of the private docker registry
- Opera (generic): create a secret from the local file, directory or literal value (literal). The corresponding API object is Opera ([o) ʊˈ pe ɪ k] ), translated as "Opaque", which is equivalent to "sensitive data".
- kubernetes.io/tls: create a TLS secret.
- kubernetes.io/service-account-token: used to be referenced by serviceaccount. When the serviceaccout is created, kubernetes will create the corresponding secret by default. If the Pod uses serviceaccount, the corresponding secret will be automatically mounted in the Pod directory / run/secrets/ kubernetes.io/serviceaccount.
Secret diagram
3) Create secret
Once you know about secret, you can start creating it. After the self signed certificate is generated, you can use the certificate key and certificate key to create a secret. Here, for convenience, I use the certificate issued by a formal organization, as shown below:
$ cd /opt/ingress $ ls tls/
1. Create a secret from the command line:
kubectl create secret tls ${CERT_NAME} --key ${KEY_FILE} --cert ${CERT_FILE}
Examples are as follows:
$ kubectl create secret tls tls-secret --key tls/tls.key --cert tls/tls.crt $ kubectl get secret
2. Create secret from yarml file
yaml file template
apiVersion: v1 kind: Secret metadata: name: testsecret-tls namespace: dashboard data: tls.crt: base64 Coded cert tls.key: base64 Coded key type: kubernetes.io/tls
Example:
Get base64 encoded cert and base64 encoded key
$ cd /opt/secret $ cat tls/tls.crt | base64 > tls/base64.tls.crt $ cat tls/tls.key | base64 > tls/base64.tls.key
If there is a problem with the creation of this method, the following errors will be reported, which will be studied later.
4) Create deployment, service, and ingress resources
1) Switch to the corresponding directory
$ cd /opt/ingress
2) Create command space
$ cat << EOF > ingress-namespace.yaml apiVersion: v1 kind: Namespace metadata: name: ingress-test EOF $ kubectl apply -f ingress-namespace.yaml
3) Create secret
$ kubectl create secret tls tls-secret --key tls/tls.key --cert tls/tls.crt -n ingress-test $ kubectl get secret -n ingress-test
4) Create deployment, service
$ cat << EOF > create-deployment-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
namespace: ingress-test
spec:
replicas: 1
selector:
matchLabels:
app: test-web
template:
metadata:
labels:
app: test-web
spec:
containers:
- name: nginx
image: nginx:1.17.1
---
apiVersion: v1 #Create a service and associate the above deployment
kind: Service
metadata:
name: web-svc
namespace: ingress-test
spec:
selector:
app: test-web
ports:
- protocol: TCP
port: 80
targetPort: 80
EOF
$ kubectl apply -f create-deployment-service.yaml
5) Create Ingress
$ cat << EOF > test-ingress.yaml
apiVersion: networking.k8s.io/v1 #Creating an ingress rule
kind: Ingress
metadata:
name: test-ingress
namespace: ingress-test
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
tls: #Issue certificate for domain name
- hosts:
- ingress.nginx.com
secretName: tls-secret
rules:
- host: ingress.nginx.com
http: #Note that the field here is http and https is not supported
paths:
- path: /index.html
pathType: Prefix
backend:
service:
name: web-svc
port:
number: 80
EOF
$ kubectl apply -f test-ingress.yaml
$ kubectl get service -n ingress-test
$ kubectl describe ingress test-ingress -n ingress-test
6) Access the nginx service through port 443 of the ingress agent
Because we have not set up a dns server, we need to bind the new domain name in the hosts file (the address can be bound to any node in the cluster)
$ kubectl get ingress -n ingress-test
#Bind the domain name in the hosts file:
192.168.0.114 ingress.nginx.com
7) https authentication
# View ingress nginx $ kubectl get svc -n ingress-nginx # Direct access, - k: allow curl to use non secure ssl connection and transfer data (certificate is not trusted) $ curl -I -k https://ingress.nginx.com/ # Access using nodePort $ curl -I -k https://ingress.nginx.com/31125
Browser access verification, https://ingress.nginx.com/
