EMQ X Team provides Helm chart for users to deploy EMQ X on kubernetes cluster with one click MQTT server This is the most recommended method of deploying EMQ X MQTT server on kubernetes or k3s cluster by EMQ X Team. This paper will use the method of handwritten yaml file to deploy a K8S cluster of EMQ X MQTT server from scratch, and analyze the details and skills in the deployment, so as to facilitate users to use flexibly in the actual deployment.
Reading this article requires users to understand the basic concepts of kubernetes and have an operational kubernetes cluster.
Deploying a single EMQ X MQTT server node on K8S
Deploying EMQ X Broker directly with Pod
In Kubernetes, the smallest management elements are not individual containers, but Pod Pod is the basic unit of execution of Kubernetes applications, that is, it is the smallest and simplest unit created or deployed in the Kubernetes object model. Pod means in the colony The process running on.
EMQ X Broker in docker hub It is convenient to deploy EMQ X Broker on a single Pod. Use the kubectl run command to create a Pod running EMQ X Broker
$ kubectl run emqx --image=emqx/emqx:v4.1-rc.1 --generator=run-pod/v1 pod/emqx created
To view the status of EMQ X Broker:
$ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE emqx 1/1 Running 0 3m13s $ kubectl exec emqx -- emqx_ctl status Node 'emqx@192.168.77.108' is started emqx 4.1-rc.1 is running
Delete Pod:
$ kubectl delete pods emqx pod "emqx" deleted
Pod is not designed as a persistent resource. It will not survive scheduling failure, node crash, or other recycling (such as due to lack of resources or other maintenance). Therefore, a controller is needed to manage the pod.
Deploying Pod with Deoloyment
Deployment A declarative method is provided for Pod and ReplicaSet to replace the previous methods ReplicationController To facilitate the management of applications. Typical application scenarios include:
- Define Deployment to create pods and replicasets
- Rolling upgrade and roll back application
- Expansion and reduction
- Pause and resume Deployment
Deploy an EMQ X Broker Pod with Deployment:
-
Define Deployment:
$ cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: emqx-deployment labels: app: emqx spec: replicas: 1 selector: matchLabels: app: emqx template: metadata: labels: app: emqx spec: containers: - name: emqx image: emqx/emqx:v4.1-rc.1 ports: - name: mqtt containerPort: 1883 - name: mqttssl containerPort: 8883 - name: mgmt containerPort: 8081 - name: ws containerPort: 8083 - name: wss containerPort: 8084 - name: dashboard containerPort: 18083 -
Deployment:
$ kubectl apply -f deployment.yaml deployment.apps/emqx-deployment created
-
To view the deployment:
$ kubectl get deployment NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/emqx-deployment 3/3 3 3 74s $ kubectl get pods NAME READY STATUS RESTARTS AGE pod/emqx-deployment-7c44dbd68-8j77l 1/1 Running 0 74s $ kubectl exec pod/emqx-deployment-7c44dbd68-8j77l -- emqx_ctl status Node 'emqx-deployment-7c44dbd68-8j77l@192.168.77.117' is started emqx 4.1-rc.1 is running
-
Try to delete Pod manually
$ kubectl delete pods emqx-deployment-7c44dbd68-8j77l pod "emqx-deployment-7c44dbd68-8j77l" deleted $ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-deployment-68fcb4bfd6-2nhh6 1/1 Running 0 59s
The output results show that the EMQ X Broker Pod is successfully deployed with Deployment. Even if the Pod is terminated unexpectedly, Deployment will create a new Pod again.
Exposing EMQ X Broker Pod service with Services
Kubernetes Pods There is a life cycle. They can be created, and destruction will not start again. If used Deployment To run the application, it can dynamically create and destroy pods.
Each Pod has its own IP address, but in Deployment, the set of pods that run at the same time may be different from the set of pods that run the application later.
This leads to a problem: if the EMQ X Broker Pod is used to serve MQTT clients, how should clients find and track the IP addresses to connect to so that clients can use the EMQ X Broker service?
The answer is: Service
Service will run in a group Pods Applications on are exposed as abstract methods for network services.
Using Service to expose EMQ X Broker Pod as network Service:
-
Define Service:
$cat service.yaml apiVersion: v1 kind: Service metadata: name: emqx-service spec: selector: app: emqx ports: - name: mqtt port: 1883 protocol: TCP targetPort: mqtt - name: mqttssl port: 8883 protocol: TCP targetPort: mqttssl - name: mgmt port: 8081 protocol: TCP targetPort: mgmt - name: ws port: 8083 protocol: TCP targetPort: ws - name: wss port: 8084 protocol: TCP targetPort: wss - name: dashboard port: 18083 protocol: TCP targetPort: dashboard -
Deploy Service:
$ kubectl apply -f service.yaml service/emqx-service created
-
View deployment
$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE emqx-service ClusterIP 10.96.54.205 <none> 1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP 58s
-
Use the IP provided by Service to view the API of EMQ X Broker
$ curl 10.96.54.205:8081/status Node emqx-deployment-68fcb4bfd6-2nhh6@192.168.77.120 is started emqx is running
At this point, a single EMQ X Broker node is deployed on kubernetes, and the EMQ X Broker Pod is managed through Deployment, and the EMQ X Broker Service is exposed through Service.
Cluster EMQ X MQTT server automatically through kubernetes
As mentioned above, a single EMQ X Broker Pod is deployed through Deployment. It is very convenient to expand the number of pods through Deployment. Execute kubectl scale Deployment ${Deployment_ Name} -- replicas ${number} command can expand the number of pods. Next, expand EMQ x broker pods to 3:
$ kubectl scale deployment emqx-deployment --replicas 3
deployment.apps/emqx-deployment scaled
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
emqx-deployment-68fcb4bfd6-2nhh6 1/1 Running 0 18m
emqx-deployment-68fcb4bfd6-mpvch 1/1 Running 0 6s
emqx-deployment-68fcb4bfd6-mx55q 1/1 Running 0 6s
$ kubectl exec emqx-deployment-68fcb4bfd6-2nhh6 -- emqx_ctl status
Node 'emqx-deployment-68fcb4bfd6-2nhh6@192.168.77.120' is started
emqx 4.1-rc.1 is running
$ kubectl exec emqx-deployment-68fcb4bfd6-2nhh6 -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
['emqx-deployment-68fcb4bfd6-2nhh6@192.168.77.120'],
stopped_nodes => []}You can see that the number of EMQ x broker pods has been expanded to three, but each Pod is independent and there is no cluster. Next, try to cluster EMQ x broker pods automatically through kubernetes.
Modify the configuration of EMQ X Broker
Check the EMQ X Broker documentation for Automatic clustering You can see that the configuration of EMQ X Broker needs to be modified
cluster.discovery = kubernetes cluster.kubernetes.apiserver = http://10.110.111.204:8080 cluster.kubernetes.service_name = ekka cluster.kubernetes.address_type = ip cluster.kubernetes.app_name = ekka
Among them cluster.kubernetes.apiserver Is the address of kubernetes apiserver, which can be obtained through the kubectl cluster info command, cluster.kubernetes.service_name is the name of the Service mentioned above, cluster.kubernetes.app_ The name is EMQ X Broker's node.name Therefore, it is also necessary to set the EMQ X Broker in the cluster to unified node.name Prefix of.
The docker image of EMQ X Broker provides the function of modifying the configuration through environment variables. You can view the details docker hub or Github.
-
Modify the yaml file of Deployment and add environment variables
$ cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: emqx-deployment labels: app: emqx spec: replicas: 3 selector: matchLabels: app: emqx template: metadata: labels: app: emqx spec: containers: - name: emqx image: emqx/emqx:v4.1-rc.1 ports: - name: mqtt containerPort: 1883 - name: mqttssl containerPort: 8883 - name: mgmt containerPort: 8081 - name: ws containerPort: 8083 - name: wss containerPort: 8084 - name: dashboard containerPort: 18083 env: - name: EMQX_NAME value: emqx - name: EMQX_CLUSTER__DISCOVERY value: k8s - name: EMQX_CLUSTER__K8S__APP_NAME value: emqx - name: EMQX_CLUSTER__K8S__SERVICE_NAME value: emqx-service - name: EMQX_CLUSTER__K8S__APISERVER value: "https://kubernetes.default.svc:443" - name: EMQX_CLUSTER__K8S__NAMESPACE value: defaultBecause ` kubectl scale deployment ${deployment_ The name} -- replicas ${number} command does not modify yaml files, so you need to set the spec.replicas: 3 .
The DNS rule of kubernetes is built into Pod, so https://kubernetes.default.svc:443 will be resolved to the address of kubernetes apiserver.
-
Delete the previous Deployment and re deploy:
$ kubectl delete deployment emqx-deployment deployment.apps "emqx-deployment" deleted $ kubectl apply -f deployment.yaml deployment.apps/emqx-deployment created
Granting Pod access to kubernetes apiserver
After deploying Deployment, check the status of the EMQ X Broker. You can see that although the EMQ X Broker is successfully started, the cluster is still not successful. Check the log of the EMQ X Broker Pod:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
emqx-deployment-5c8cfc4d75-67lmt 1/1 Running 0 5s
emqx-deployment-5c8cfc4d75-r6jgb 1/1 Running 0 5s
emqx-deployment-5c8cfc4d75-wv2hj 1/1 Running 0 5s
$ kubectl exec emqx-deployment-5c8cfc4d75-67lmt -- emqx_ctl status
Node 'emqx@192.168.87.150' is started
emqx 4.1-rc.1 is running
$ kubectl exec emqx-deployment-5c8cfc4d75-67lmt -- emqx_ctl cluster status
Cluster status: #{running_nodes => ['emqx@192.168.87.150'],
stopped_nodes => []}
$ kubectl logs emqx-deployment-76f6895c46-4684f
···
(emqx@192.168.87.150)1> 2020-05-20 01:48:39.726 [error] Ekka(AutoCluster): Discovery error: {403,
"{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"endpoints \\\"emqx-service\\\" is forbidden: User \\\"system:serviceaccount:default:default\\\" cannot get resource \\\"endpoints\\\" in API group \\\"\\\" in the namespace \\\"default\\\"\",\"reason\":\"Forbidden\",\"details\":{\"name\":\"emqx-service\",\"kind\":\"endpoints\"},\"code\":403}\n"}
···Pod was denied access to kubernetes apiserver due to permission problem, and returned HTTP 403, so the cluster failed.
Ordinary Pod can't access kubernetes apiserver. There are two ways to solve this problem. One is to open the http interface of kubernetes apiserver, but this method has some security risks. The other is to configure RBAC authentication through ServiceAccount, Role and RoleBinding.
-
Define ServiceAccount, Role, and RoleBinding:
$ cat rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: namespace: default name: emqx --- kind: Role apiVersion: rbac.authorization.kubernetes.io/v1beta1 metadata: namespace: default name: emqx rules: - apiGroups: - "" resources: - endpoints verbs: - get - watch - list --- kind: RoleBinding apiVersion: rbac.authorization.kubernetes.io/v1beta1 metadata: namespace: default name: emqx subjects: - kind: ServiceAccount name: emqx namespace: default roleRef: kind: Role name: emqx apiGroup: rbac.authorization.kubernetes.io
-
Deploy the appropriate resources:
$ kubectl apply -f rbac.yaml serviceaccount/emqx created role.rbac.authorization.kubernetes.io/emqx created rolebinding.rbac.authorization.kubernetes.io/emqx created
-
Modify the yaml file of Deployment and add spec.template.spec.serviceAccountName and redeploy:
$cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: emqx-deployment labels: app: emqx spec: replicas: 3 selector: matchLabels: app: emqx template: metadata: labels: app: emqx spec: serviceAccountName: emqx containers: - name: emqx image: emqx/emqx:v4.1-rc.1 ports: - name: mqtt containerPort: 1883 - name: mqttssl containerPort: 8883 - name: mgmt containerPort: 8081 - name: ws containerPort: 8083 - name: wss containerPort: 8084 - name: dashboard containerPort: 18083 env: - name: EMQX_NAME value: emqx - name: EMQX_CLUSTER__DISCOVERY value: kubernetes - name: EMQX_CLUSTER__K8S__APP_NAME value: emqx - name: EMQX_CLUSTER__K8S__SERVICE_NAME value: emqx-service - name: EMQX_CLUSTER__K8S__APISERVER value: "https://kubernetes.default.svc:443" - name: EMQX_CLUSTER__K8S__NAMESPACE value: default $ kubectl delete deployment emqx-deployment deployment.apps "emqx-deployment" deleted $ kubectl apply -f deployment.yaml deployment.apps/emqx-deployment created -
View status:
$ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-deployment-6b854486c-dhd7p 1/1 Running 0 10s emqx-deployment-6b854486c-psv2r 1/1 Running 0 10s emqx-deployment-6b854486c-tdzld 1/1 Running 0 10s $ kubectl exec emqx-deployment-6b854486c-dhd7p -- emqx_ctl status Node 'emqx@192.168.77.92' is started emqx 4.1-rc.1 is running $ kubectl exec emqx-deployment-6b854486c-dhd7p -- emqx_ctl cluster status Cluster status: #{running_nodes => ['emqx@192.168.77.115','emqx@192.168.77.92', 'emqx@192.168.87.157'], stopped_nodes => []} -
Abort a Pod:
$ kubectl delete pods emqx-deployment-6b854486c-dhd7p pod "emqx-deployment-6b854486c-dhd7p" deleted $ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-deployment-6b854486c-846v7 1/1 Running 0 56s emqx-deployment-6b854486c-psv2r 1/1 Running 0 3m50s emqx-deployment-6b854486c-tdzld 1/1 Running 0 3m50s $ kubectl exec emqx-deployment-6b854486c-846v7 -- emqx_ctl cluster status Cluster status: #{running_nodes => ['emqx@192.168.77.115','emqx@192.168.77.84', 'emqx@192.168.87.157'], stopped_nodes => ['emqx@192.168.77.92']}The output results show that EMQ X Broker will display the stopped pods correctly and add the newly created Pod of Deployment to the cluster.
So far, EMQ X Broker successfully established the cluster on kubernetes.
Persistent EMQ X Broker cluster
The Deployment used above is used to manage the Pod, but the network of the Pod is constantly changing. Moreover, when the Pod is destroyed and rebuilt, the data and configuration stored in the EMQ X Broker will disappear. This is unacceptable in production. Next, try to persist the cluster of the EMQ X Broker. Even if the Pod is destroyed and rebuilt, the EMQ X Broker will be destroyed The data can still be saved.
ConfigMap
ConfigMap configMap is an API object used to save non confidential data to key value pairs. When used, it can be used as an environment variable, a command-line parameter, or a configuration file in a storage volume.
ConfigMap combines your environment configuration information with Container image Decoupling, easy to modify the application configuration.
ConfigMap does not provide privacy or encryption. If the data you want to store is confidential, use Secret , or use other third-party tools to keep your data private, rather than using ConfigMap.
Next, use ConfigMap to record the configuration of EMQ X Broker and import them into Deployment as environment variables.
-
Define Configmap and deploy:
$cat configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: emqx-config data: EMQX_CLUSTER__K8S__ADDRESS_TYPE: "hostname" EMQX_CLUSTER__K8S__APISERVER: "https://kubernetes.default.svc:443" EMQX_CLUSTER__K8S__SUFFIX: "svc.cluster.local" $ kubectl apply -f configmap.yaml configmap/emqx-config created
-
Configure Deployment to use Configmap
$cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: emqx-deployment labels: app: emqx spec: replicas: 3 selector: matchLabels: app: emqx template: metadata: labels: app: emqx spec: serviceAccountName: emqx containers: - name: emqx image: emqx/emqx:v4.1-rc.1 ports: - name: mqtt containerPort: 1883 - name: mqttssl containerPort: 8883 - name: mgmt containerPort: 8081 - name: ws containerPort: 8083 - name: wss containerPort: 8084 - name: dashboard containerPort: 18083 envFrom: - configMapRef: name: emqx-config -
Redeploy the Deployment and view the status
$ kubectl delete -f deployment.yaml deployment.apps "emqx-deployment" deleted $ kubectl apply -f deployment.yaml deployment.apps/emqx-deployment created $ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-deployment-5c7696b5d7-k9lzj 1/1 Running 0 3s emqx-deployment-5c7696b5d7-mdwkt 1/1 Running 0 3s emqx-deployment-5c7696b5d7-z57z7 1/1 Running 0 3s $ kubectl exec emqx-deployment-5c7696b5d7-k9lzj -- emqx_ctl status Node 'emqx@192.168.87.149' is started emqx 4.1-rc.1 is running $ kubectl exec emqx-deployment-5c7696b5d7-k9lzj -- emqx_ctl cluster status Cluster status: #{running_nodes => ['emqx@192.168.77.106','emqx@192.168.77.107', 'emqx@192.168.87.149'], stopped_nodes => []}
The configuration file of EMQ X Broker has been decoupled into Configmap. If necessary, you can configure one or more configmaps freely and import them into Pod as environment variables or files.
StatefulSet
StatefulSet It is designed to solve the problem of state and no state for applications
- Stable persistent storage, that is, the same persistent data can be accessed after Pod rescheduling, which is based on PVC
- The stable network flag, that is, after the Pod is rescheduled, its PodName and HostName remain unchanged, which is based on the Headless Service (i.e. the Service without Cluster IP)
- Orderly deployment, orderly expansion, that is, pods are sequential. When deploying or expanding, they should be carried out in sequence according to the defined order (i.e. from 0 to N-1, all previous pods must be in Running and Ready status before the next Pod is run), which is implemented based on init containers
- Orderly shrink, orderly delete (from N-1 to 0)
From the above application scenario, we can see that StatefulSet is composed of the following parts:
- Headless Service for defining DNS domain
- volumeClaimTemplates for creating PersistentVolumes
- Define StatefulSet for specific application
The DNS format of each Pod in StatefulSet is statefulsetname - {0.. n-1} serviceName.namespace.svc . cluster.local In which
- serviceName is the name of the Headless Service
- 0..N-1 is the sequence number of Pod, starting from 0 to N-1
- statefulSetName is the name of StatefulSet
- The namespace is the namespace where the service is located, and the headless service and stateful set must be in the same namespace
- . cluster.local Cluster Domain
Next, use StatefulSet instead of Deployment to manage pods.
-
Delete Deployment:
$ kubectl delete deployment emqx-deployment deployment.apps "emqx-deployment" deleted
-
Define StatefulSet:
$cat statefulset.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: emqx-statefulset labels: app: emqx spec: serviceName: emqx-headless updateStrategy: type: RollingUpdate replicas: 3 selector: matchLabels: app: emqx template: metadata: labels: app: emqx spec: serviceAccountName: emqx containers: - name: emqx image: emqx/emqx:v4.1-rc.1 ports: - name: mqtt containerPort: 1883 - name: mqttssl containerPort: 8883 - name: mgmt containerPort: 8081 - name: ws containerPort: 8083 - name: wss containerPort: 8084 - name: dashboard containerPort: 18083 envFrom: - configMapRef: name: emqx-configNote that stateful set requires Headless Service to implement stable network flag, so a Service needs to be defined
$cat headless.yaml apiVersion: v1 kind: Service metadata: name: emqx-headless spec: type: ClusterIP clusterIP: None selector: app: emqx ports: - name: mqtt port: 1883 protocol: TCP targetPort: 1883 - name: mqttssl port: 8883 protocol: TCP targetPort: 8883 - name: mgmt port: 8081 protocol: TCP targetPort: 8081 - name: websocket port: 8083 protocol: TCP targetPort: 8083 - name: wss port: 8084 protocol: TCP targetPort: 8084 - name: dashboard port: 18083 protocol: TCP targetPort: 18083
Because the Headless Service does not require IP, clusterIP: None is configured.
-
Deploy the appropriate resources:
$ kubectl apply -f headless-service.yaml service/emqx-headless created $ kubectl apply -f statefulset.yaml statefulset.apps/emqx-deployment created $ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-statefulset-0 1/1 Running 0 2m59s emqx-statefulset-1 1/1 Running 0 2m57s emqx-statefulset-2 1/1 Running 0 2m54s $ kubectl exec emqx-statefulset-0 -- emqx_ctl cluster status Cluster status: #{running_nodes => ['emqx@192.168.77.105','emqx@192.168.87.153', 'emqx@192.168.87.155'], stopped_nodes => []} -
Update Configmap:
StatefulSet provides a stable network flag. EMQ X Broker supports the use of hostname and dns rules to implement cluster on behalf of IP. Taking hostname as an example, it needs to be modified emqx.conf :
cluster.kubernetes.address_type = hostname cluster.kubernetes.suffix = "svc.cluster.local"
The DNS rules of Pod in kubernetes cluster can be customized by users. EMQ X Broker provides cluster.kubernetes.suffix This paper uses the default DNS rule: statefulsetname - {0.. n-1} serviceName.namespace.svc . cluster.local , the servicename in the DNS rule is the headless used by StatefulSet Service, so you also need to cluster.kubernetes.service_name is changed to Headless Service Name.
To convert a configuration item into an environment variable, you need to configure it in Configmap
EMQX_CLUSTER__K8S__ADDRESS_TYPE: "hostname" EMQX_CLUSTER__K8S__SUFFIX: "svc.cluster.local" EMQX_CLUSTER__K8S__SERVICE_NAME: emqx-headless
Configmap provides the hot update function. Execute $kubectl edit configmap emqx config to hot update configmap.
-
Redeploy StatefulSet:
After the Configmap update, the Pod will not restart. We need to manually update the StatefulSet
$ kubectl delete statefulset emqx-statefulset statefulset.apps "emqx-statefulset" deleted $ kubectl apply -f statefulset.yaml statefulset.apps/emqx-statefulset created $ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-statefulset-0 1/1 Running 0 115s emqx-statefulset-1 1/1 Running 0 112s emqx-statefulset-2 1/1 Running 0 110s $ kubectl exec emqx-statefulset-2 -- emqx_ctl cluster status Cluster status: #{running_nodes => ['emqx@emqx-statefulset-0.emqx-headless.default.svc.cluster.local', 'emqx@emqx-statefulset-1.emqx-headless.default.svc.cluster.local', 'emqx@emqx-statefulset-2.emqx-headless.default.svc.cluster.local'], stopped_nodes => []}You can see that the new EMQ X Broker cluster has been successfully established.
-
Abort a Pod:
After the Pod in StatefulSet is rescheduled, its PodName and HostName remain unchanged. Here's a try:
$ kubectl get pods kuNAME READY STATUS RESTARTS AGE emqx-statefulset-0 1/1 Running 0 6m20s emqx-statefulset-1 1/1 Running 0 6m17s emqx-statefulset-2 1/1 Running 0 6m15s $ kubectl delete pod emqx-statefulset-0 pod "emqx-statefulset-0" deleted $ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-statefulset-0 1/1 Running 0 27s emqx-statefulset-1 1/1 Running 0 9m45s emqx-statefulset-2 1/1 Running 0 9m43s $ kubectl exec emqx-statefulset-2 -- emqx_ctl cluster status Cluster status: #{running_nodes => ['emqx@emqx-statefulset-0.emqx-headless.default.svc.cluster.local', 'emqx@emqx-statefulset-1.emqx-headless.default.svc.cluster.local', 'emqx@emqx-statefulset-2.emqx-headless.default.svc.cluster.local'], stopped_nodes => []}As expected, StatefulSet reschedules a Pod with the same network flag, and the EMQ X Broker in the Pod successfully joins the cluster.
StorageClasses, PersistentVolume, and PersistentVolumeClaim
A persistent Volume (PV) is a storage set up by an administrator and is part of a cluster. Just as nodes are resources in a cluster, PV is also a resource in a cluster. PV is a Volume plug-in such as Volume, but has a lifecycle independent of the Pod that uses PV. This API object contains the details of the storage implementation, that is, NFS, iSCSI, or cloud vendor specific storage systems.
Persistent volume claim (PVC) is a user stored request. It's similar to pod. Pod consumes node resources and PVC consumes PV resources. Pods can request specific levels of resources (CPU and memory). Declarations can request specific sizes and access modes (for example, they can be mounted in read / write once or read-only multiple times).
StorageClass provides administrators with a way to describe a storage "class.". Different classes may map to different quality of service levels or backup policies, or to any policy determined by the Cluster Administrator. Kubernetes itself does not know what the various classes represent. This concept is sometimes referred to as a "configuration file" in other storage systems.
When deploying EMQ X Broker, PV or StorageClass can be created in advance, and then the directory / opt/emqx/data/mnesia of EMQ X Broker can be mounted by using PVC. After Pods are rescheduled, EMQ X will obtain data from / opt/emqx/data/mnesia directory and recover, so as to realize the persistence of EMQ X Broker.
-
Define StatefulSet
$cat statefulset.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: emqx-statefulset labels: app: emqx spec: replicas: 3 serviceName: emqx-headless updateStrategy: type: RollingUpdate selector: matchLabels: app: emqx template: metadata: labels: app: emqx spec: volumes: - name: emqx-data persistentVolumeClaim: claimName: emqx-pvc serviceAccountName: emqx containers: - name: emqx image: emqx/emqx:v4.1-rc.1 ports: - name: mqtt containerPort: 1883 - name: mqttssl containerPort: 8883 - name: mgmt containerPort: 8081 - name: ws containerPort: 8083 - name: wss containerPort: 8084 - name: dashboard containerPort: 18083 envFrom: - configMapRef: name: emqx-config volumeMounts: - name: emqx-data mountPath: "/opt/emqx/data/mnesia" volumeClaimTemplates: - metadata: name: emqx-pvc annotations: volume.alpha.kubernetes.io/storage-class: manual spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1GiThe file first specifies the storage class with the name of StorageClass as manual through volumeClaimTemplates to create a PVC resource named emqx PVC. The read-write mode of the PVC resource is ReadWriteOnce, which requires 1Gi of space. Then, the PVC is defined as the volumes with the name of emqx data, and the volumes are mounted in the / opt/emqx/data/mnesia directory in Pod.
-
Deployment resources:
Before deploying StatefulSet, users or kubernetes cluster administrators need to create their own storage classes.
$ kubectl apply -f statefulset.yaml statefulset.apps/emqx-statefulset created $ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-statefulset-0 1/1 Running 0 27s emqx-statefulset-1 1/1 Running 0 9m45s emqx-statefulset-2 1/1 Running 0 9m43s $ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE emqx-data-emqx-statefulset-0 Bound pvc-8094cd75-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 2m11s emqx-data-emqx-statefulset-0 Bound pvc-9325441d-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 99s emqx-data-emqx-statefulset-0 Bound pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 56s
The output results show that the state of the PVC is Bound, and the PVC storage has been successfully established. When the Pod is rescheduled, EMQ X Broker will read the data mounted in the PVC to achieve persistence.
EMQ X Broker establishes a persistent cluster on kubernetes. This article omits some details. The deployment process is also a simple Demo. Users can read it by themselves kubernetes documentation With EMQ X Team Helm chart source code To continue in-depth research, of course, we welcome the contribution of issue, pull requests and start in Github.