Rapid deployment of EMQ X clusters on Kubernetes through Helm

Keywords: Kubernetes github SSL Google

Provided through EMQ X Helm chart EMQ X can be easily deployed to Kubernetes Platform. Before you start, make sure you follow Helm Document Guide To install Helm.

On Kubernetes

Kubernetes is a widely used open source container orchestration platform for automatic deployment, expansion and operation of applications.

With Kubernetes, it can quickly and efficiently respond to customer needs:

  • Fast and predictable deployment of applications
  • Ability to extend applications in real time
  • Seamlessly release new functionality without affecting existing business
  • Optimizing Hardware Resources and Reducing Costs

Kubernetes has the following characteristics:

  • Portability: Full support for public, private, hybrid or multi-cloud architectures
  • Extensibility: It is modular, pluggable, mountable, composable, and supports various forms of extension
  • Self-repairing: It can self-maintain application status, self-restart, self-replicate, self-scaling, and provides powerful self-repairing ability through declarative grammar.

The Kubernetes project was launched by Google in 2014. Kubernetes was founded Based on more than a decade of experience in operations and maintenance, all of Google's applications run on containers. Combined with the best ideas and practices in the community, it may be the most popular container platform.

About Helm

Helm It is a package management tool for kubernetes applications to manage [charts]( https://github.com/kubernetes/charts )—— The pre-configured installation package resources are somewhat similar to the yum in Ubuntu's APT and entOS.

Helm chart is a yaml file used to encapsulate the native application of kubernetes. It can customize some metadata of the application when deploying the application, and then distribute it with the application.

The main roles of Helm and charts are:

  • Application Encapsulation
  • version management
  • Dependency checking
  • Facilitate application distribution

Setup

  • Kubernetes 1.6+
  • Helm

Rapid deployment of a simple EMQ X cluster

  • Add Helm Warehouse

    $ helm repo add emqx https://repos.emqx.io/charts
    
  • Query EMQ X

    helm search emqx
    NAME     	CHART VERSION	APP VERSION  	DESCRIPTION
    emqx/emqx	v1.0.0-beta.1	v1.0.0-beta.1	A Helm chart for Emqx
    

    EMQ X Chart is currently Beta version, so you need to add devel when executing the helm install command. The official version of Chart does not need to add this option, please note.

  • Start the EMQ X cluster, set service.type=NodePort, and specify emqx/emqx:v3.2.3 as the image to be used

    $ helm install --devel --name my-emqx --set service.type=NodePort --set image=emqx/emqx:v3.2.3 emqx/emqx
    
  • View EMQ X Cluster

    $ kubectl get pods
    NAME       READY  STATUS             RESTARTS  AGE
    my-emqx-0  1/1    Running            0         81m
    my-emqx-1  1/1    Running            0         81m
    my-emqx-2  1/1    Running            0         80m
    
    $ kubectl exec -it my-emqx-0 -- emqx_ctl cluster status
    Cluster status: [{running_nodes,['my-emqx@192.168.109.82',
                                     'my-emqx@192.168.71.78',
                                     'my-emqx@192.168.95.126']}]
    
  • View EMQ X service

    $ kubectl get svc
    NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                                      AGE
    kubernetes           ClusterIP   10.100.0.1      <none>        443/TCP                                                                                      7h26m
    my-emqx              NodePort    10.100.213.62   <none>        1883:30068/TCP,8883:32463/TCP,8080:31854/TCP,8083:32057/TCP,8084:31459/TCP,18083:31539/TCP   57s
    my-emqx-headless     ClusterIP   None            <none>        1883/TCP,8883/TCP,8080/TCP,8083/TCP,8084/TCP,18083/TCP                                       57s
    

    You can see that the host IP of my-emqx port 18083 is 31539. (NodePort changes at each deployment, depending on the actual deployment.)

  • Access port 3159 of any node IP of Kubernetes, enter default user name: admin, default password: public, login EMQ X Dashboard.

  • Delete EMQ X cluster

    $ helm del --purge my-emqx
    

Deploy a persistent EMQ X cluster

EMQ X implements persistent pods by creating a PVC resource mount/opt/emqx/data/mnesia directory. Before deploying EMQX, users need to deploy load balancers such as Haproxy or Nginx-PLUS, and create PV resources or Storage Classes resources in Kubernetes by themselves.

  • Add Helm Warehouse

    $ helm repo add emqx https://repos.emqx.io/charts
    
  • Start EMQ X cluster

    Persistent EMQ X clusters require the following settings:

    image=emqx/emqx:v3.2.3
    emqxAddressType=hostname
    persistence.enabled=true
    

    ** Note: ** Persistent EMQ X only supports versions above v3.2.1

    • If the user deploys PV resources, set persistence.existingClaim=your_pv_name

      $ helm install --devel --name my-emqx --set image=emqx/emqx:v3.2.3 --set emqxAddressType=hostname --set persistence.enabled=true --set persistence.existingClaim=your_pv_name  emqx/emqx
      
    • If the user deploys Storage Classes resources, set persistence.storageClass=your_storageClass_name

      $ helm install --devel --name my-emqx --set image=emqx/emqx:v3.2.3 --set emqxAddressType=hostname --set persistence.enabled=true --set persistence.storageClass=your_storageClass_name emqx/emqx
      
  • View EMQ X Cluster

    $ kubectl get pods
    NAME       READY  STATUS             RESTARTS  AGE
    my-emqx-0  1/1    Running            0         81m
    my-emqx-1  1/1    Running            0         81m
    my-emqx-2  1/1    Running            0         80m
    
    $ kubectl exec -it my-emqx-0 -- emqx_ctl cluster status
    Cluster status: [{running_nodes,['my-emqx@my-emqx-2.my-emqx-headless.default.svc.cluster.local',
                                     'my-emqx@my-emqx-1.my-emqx-headless.default.svc.cluster.local',
                                     'my-emqx@my-emqx-0.my-emqx-headless.default.svc.cluster.local']}]
    
  • Taking Storage Classes as an example, we can see that PVC resources have been successfully established.

    $ kubectl get pvc
    NAME                  STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    emqx-data-my-emqx-0   Bound     pvc-8094cd75-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            2m11s
    emqx-data-my-emqx-1   Bound     pvc-9325441d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            99s
    emqx-data-my-emqx-2   Bound     pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            56s
    

    Clusters mount EMQ X's / opt/emqx/data/mnesia directory into PVC. When Pods are rescheduled, EMQX retrieves data from the / opt/emqx/data/mnesia directory and restores it.

  • View lusterIP of EMQ X

    $ kubectl get svc
    NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                  AGE
    kubernetes           ClusterIP   10.100.0.1      <none>        443/TCP                                                  24h
    my-emqx              ClusterIP   10.100.205.13   <none>        1883/TCP,8883/TCP,8080/TCP,8083/TCP,8084/TCP,18083/TCP   26m
    my-emqx-headless     ClusterIP   None            <none>        1883/TCP,8883/TCP,8080/TCP,8083/TCP,8084/TCP,18083/TCP   26m
    

    You can see that my-emqx's Cluster IP is 10.100.205.13 (Cluster IP changes at each deployment, depending on the actual deployment).

  • Forward the 1883, 8883, 8080, 8083, 8084, 18083 ports of the load balancing monitor URL s to my-emqx lusterIP. If there is a need for TLS connection, it is recommended to terminate the SSL connection in the load balancer. TLS secure connection between client and load balancer, and ordinary TCP connection between LB and EMQ X.

  • Access URL:18083, enter the default user name: admin, default password: public, login EMQ X dashboard.

  • Using the helm upgrade command, you can easily extend the EMQ X cluster. Here's an example of adding an EMQX node to show the helm upgrade command

    # Change the number of nodes in EMQ X to 5
    # Note: The number of nodes in EMQ X is recommended to be singular
    $ helm upgrade --set replicas=5 my-emqx emqx/emqx
    
    $ kubectl get pods
    NAME       READY  STATUS             RESTARTS  AGE
    my-emqx-0  1/1    Running            0         81m
    my-emqx-1  1/1    Running            0         81m
    my-emqx-2  1/1    Running            0         80m
    my-emqx-3  1/1    Running            0         22m
    my-emqx-4  1/1    Running            0         22m
    
    $ kubectl exec -it my-emqx-0 -- emqx_ctl cluster status
    Cluster status: [{running_nodes,['my-emqx@my-emqx-4.my-emqx-headless.default.svc.cluster.local',
                                     'my-emqx@my-emqx-3.my-emqx-headless.default.svc.cluster.local',
                                     'my-emqx@my-emqx-2.my-emqx-headless.default.svc.cluster.local',
                                     'my-emqx@my-emqx-1.my-emqx-headless.default.svc.cluster.local',
                                     'my-emqx@my-emqx-0.my-emqx-headless.default.svc.cluster.local']}]
    
  • Delete EMQ X cluster

    $ helm del --purge my-emqx
    

    ** Note: ** PVC resources will not be automatically released after EMQ X cluster deletion in order to restore EMQX, confirming that manual deletion of PVC resources is required after recovery is not required.

    $ kubectl get pvc
    NAME                  STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    emqx-data-my-emqx-0   Bound     pvc-8094cd75-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            84m
    emqx-data-my-emqx-1   Bound     pvc-9325441d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            84m
    emqx-data-my-emqx-2   Bound     pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            83m
    emqx-data-my-emqx-3   Bound     pvc-b6c5a565-adbd-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            25m
    emqx-data-my-emqx-4   Bound     pvc-c626cafd-adbd-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            25m
    
    $ kubectl delete pvc emqx-data-my-emqx-0 emqx-data-my-emqx-1 emqx-data-my-emqx-2 emqx-data-my-emqx-3 emqx-data-my-emqx-4                    
    persistentvolumeclaim "emqx-data-my-emqx-0" deleted
    persistentvolumeclaim "emqx-data-my-emqx-1" deleted
    persistentvolumeclaim "emqx-data-my-emqx-2" deleted
    persistentvolumeclaim "emqx-data-my-emqx-3" deleted
    persistentvolumeclaim "emqx-data-my-emqx-4" deleted
    

Deployment of EMQ X Edge Cluster and EMQ X Enterprise Edition Cluster

EMQ X Edge

Deploying the EMQ X Edge cluster specifies image=emqx/emqx-edge:v3.2.3, and other settings are consistent with deploying the EMQ X cluster.

$ helm install --devel --name my-emqx --set service.type=NodePort --set image=emqx/emqx-edge:v3.2.3 emqx/emqx

EMQ X EE

Deployment of EMQ X Enterprise Edition Cluster requires login first emqx.io Apply for and download the License file and create the License file as a Secret resource

$ kubectl create secret generic your-license-secret-name --from-file=/path/to/emqx.lic

Then specify emqxLicneseSecretName=your-license-secret-name, image=emqx/emqx-ee:v3.4.0 at deployment time, and other settings are consistent with deploying EMQ X clusters.

helm install --devel --name my-emqx --set service.type=NodePort --set image=emqx/emqx-ee:v3.2.3 --set emqxLicneseSecretName=your-license-secret-name emqx/emqx

EMQ X Helm Chart Configuration Item

parameter describe Default Value
apiserver Kubernates API server https://kubernetes.default.svc:443
namespace kubernetes namespace default
replicas The number of EMQ X nodes, it is recommended to keep odd number of nodes, otherwise after fissure can not be automatically restored. 3
image EMQ X Mirror Name emqx/emqx:latest
imagePullPolicy The Strategy of Obtaining Mirror Image IfNotPresent
persistence.enabled Whether to Enable PVC false
persistence.storageClass Storage class name nil
persistence.existingClaim PV name ""
persistence.accessMode PVC Access Mode ReadWriteOnce
persistence.size PVC capacity 20Mi
resources CPU/memory resources {}
service.type Emqx Cluster Service Type ClusterIP
service.mqttPort MQTT Protocol Port 1883
service.mqttsslPort MQTT/SSL Port 8883
service.mgmtPort Manage API ports 8080
service.websocketPort MQTT/WebSocket Port 8083
service.wssPort MQTT/WebSocket/SSL Port 8084
service.dashboardPort Dashboard port 18083
emqxAddressType Optional values for obtaining EMQ X node < br > from k8s service: ip, dns, hostname < br > Note: hosename only supports versions after EMQ X v3.2.1 ip
emqxConfig EMQ X Configuration Item, Details See File {}
emqxLicneseSecretName EMQ X Enterprise Edition needs to manually create License files as Secret resources ""

When complex parameters need to be set, Yaml files can be used to record parameters.

$ helm install --devel --name my-emqx -f values.yaml emqx/emqx

You can learn from Github Get the default values.yaml

For more information, please visit our official website emqx.io Or focus on our open source projects github.com/emqx/emqx For detailed documentation, please visit Official documents.

Posted by axm on Mon, 23 Sep 2019 23:53:56 -0700