1. background

Container technology, represented by Docker and Rocket, is becoming more and more popular. It changes the way companies and users create, publish and run distributed applications. It will bring its due value to the cloud computing industry in the next five years. Its attraction lies in:

1) Resource independence and isolation

Resource isolation is the most basic requirement of cloud computing platform. Docker restricts hardware resources and software running environment through Linux namespace and cgroup, isolates itself from other applications on the host, and does not affect each other. Tens of thousands of "containers" are arranged neatly on different applications or services to "ship" or "ship" unloading in units of "container". Different companies and different types of "cargo" (procedures, components, operating environment and dependence required for running applications) remain independent.

2) Environmental Consistency

After the application development, the development engineer builds a docker image. The container created based on this image is like a container, which packs all kinds of "bulk cargo" (programs, components, operating environment, dependencies needed to run the application). No matter where the container is: development environment, test environment and production environment, it can ensure that the type and number of "goods" in the container are exactly the same, the software package will not be missing in the test environment, the environment variables will not forget to configure in the production environment, and the development environment and production environment will not be abnormal because of the installation of different versions of dependence. Such consistency benefits from the fact that the "build docker image" has been sealed into "containers", and that each link is in the transport of this complete "container" that does not need to be split and merged.

3) Lightweight

Compared with the traditional virtualization technology (VM), the performance loss of using docker on cpu, memory, disk IO, network IO has the same level or even better performance. Container's rapid creation, start-up and destruction have received many praises.

4)Build Once, Run Everywhere

This feature has attracted many people. When transferring and exchanging goods (applications) between "cars", "trains", "ships" (private clouds, public clouds and other services), only "docker container s" meeting standard specifications and loading and unloading modes need to be migrated, which reduces the time-consuming manual "loading and unloading" (on-line and off-line applications) and brings huge benefits. Save time and manpower cost. This makes it possible that only a few operators will be able to operate and maintain container clusters for ultra-large-scale loading line applications in the future, just as a few machine operators will be able to load and unload a 10,000-class container ship in a few hours after the 1960s.

Container technology is now also widely used in the field of database. Its "Build Once, Run Everywhere" feature greatly reduces the time spent on installing and configuring database environments, because even for DBA s that have been working on databases for many years, installing and configuring database environments is still a seemingly simple but often not smooth job. Of course, other advantages of container technology are also well applied in the use of database.

SequoiaDB, as an excellent domestic distributed NewSQL database, has been recognized by more and more users. Taking Docker as an example, this paper focuses on how to quickly build SequoiaDB image with Dockerfile, and how to quickly build and start SequoiaDB cluster supply system using container.

1. Building SequoiaDB Mirror

How to install docker and configure mirror warehouse is not the focus of this article. There are many related technical articles on the Internet. It should be pointed out that this paper uses Aliyun Mirror Warehouse, because the speed of uploading the Mirror to Docker's official warehouse is really not complimentary. How to register and use Aliyun Mirror Warehouse, you can refer to the article( http://www.jb51.net/article/123101.htm).

STEP 1: Create a Docker file, which reads as follows: Sequoiadb DOCKERFILES PROJECT with a few simple instructions

#This is the Dockerfile for Sequoiadb 2.8.4

REQUIRED FILES TO BUILD THIS IMAGE

(1) sequoiadb-2.8.4-linux_x86_64-enterprise-installer.run

(2) installSDB.sh

HOW TO BUILD THIS IMAGE
 Put all downloaded files in the same directory as this Dockerfile
Run:$ sudo docker build -t sequoiadb:2.8.4 .
Pull base image

FROM ubuntu

Environment variables required for this build

ENV INSTALL_BIN_FILE="sequoiadb-2.8.4-linux_x86_64-enterprise-installer.run" \

    INSTALL_SDB_SCRIPT="installSDB.sh" \

    INSTALL_DIR="/opt/sequoiadb"

      Copy binaries
ADD $INSTALL_BIN_FILE $INSTALL_SDB_SCRIPT $INSTALL_DIR/

Install SDB software binaries

RUN chmod 755 $INSTALL_DIR/$INSTALL_SDB_SCRIPT \

    && $INSTALL_DIR/$INSTALL_SDB_SCRIPT \

    && rm $INSTALL_DIR/$INSTALL_SDB_SCRIPT

The installSDB.sh script is as follows:

chmod 755 $INSTALL_DIR/$INSTALL_BIN_FILE

$INSTALL_DIR/$INSTALL_BIN_FILE --mode unattended

rm $INSTALL_DIR/$INSTALL_BIN_FILE

echo 'service sdbcm start' >> /root/.bashrc

It should be noted that this example uses Sequoiadb Enterprise Version 2.8.4. You can also download the community version (select tar package, download and decompress) from Jushan's official website to replace the media name in this example.

Jushan Official Website Download Address: http://download.sequoiadb.com/cn/

STEP 2: Creating Mirrors

root user execution:

docker build -t sequoiadb:2.8.4 .

If you are a regular user, you need to use sudo:

sudo docker build -t sequoiadb:2.8.4 .

STEP3: Landing in Aliyun Mirror Warehouse

docker login --username=xxx registry.cn-hangzhou.aliyuncs.com

Among them, xxx is your account registered in Aliyun.

STEP4: View the local sequoiadb image id

docker images

STEP5: Mark the local image and place it in the Aliyun warehouse

docker tag 04dc528f2a6f registry.cn-hangzhou.aliyuncs.com/508mars/sequoiadb:latest

04dc528f2a6f is my local sequoiadb image id. The new tag format has certain requirements. Regisry. cn-hangzhou. aliyuncs. com is the warehouse address of Aliyun, 508mars is my user name in Aliyun, sequoiadb is the mirror name, latest is the tag.

STEP6: Submit sequoiadb image to mirror Library

docker push registry.cn-hangzhou.aliyuncs.com/508mars/sequoiadb:latest

1. Start SequoiaDB Cluster with Containers

Docker's network adopts bridge mode by default. Containers using bridge mode have the following characteristics:

1) Containers of the same host can ping each other

2) Containers of different hosts ping differently from each other

However, the SequoiaDB cluster requires that all nodes are interoperable, so if the container running SequoiaDB runs on different hosts, docker's default network mode is obviously inappropriate. There are many ways to solve the problem of connectivity between different host containers. This article only introduces the solution of weave virtual network, because weave also provides a DNS server function. With this function, when using containers to deploy SequoiaDB cluster, it is no longer necessary to modify the / etc/hosts inside each container, which greatly simplifies the steps of automatic deployment.

STEP1: Install weave Network

curl -s -L git.io/weave -o /usr/local/bin/weave

chmod a+x /usr/local/bin/weave

It needs to be installed on all hosts. The author uses three virtual machines as hosts: sdb1, sdb2 and sdb3.

STEP2: Start weave Network

weave launch

The weave image will be downloaded on the first boot.

STEP3: Download sequoiadb image from Aliyun warehouse

docker pull registry.cn-hangzhou.aliyuncs.com/508mars/sequoiadb

STEP4: Create docker mount volumes on all hosts

cd /home/sdbadmin

mkdir -p data/disk1 data/disk2 data/disk3

mkdir -p conf/local

chmod -R 777 data

chmod -R 777 conf

The location of mounted volumes can be customized, but generally there are two types of mounted volumes that need to be created. One is used to store aggregate data, such as data/disk1, data/disk2, data/disk3 in this example, and the other is used to store node configuration information, such as conf/local in this example. In this way, even if the container is deleted by mistake, a new container can still be activated to play the role of the container being deleted by mistake.

STEP5: Start Container

sdb1:

weave stop

weave launch

eval $(weave env)

docker run -dit --name sdbserver1 -p 11810:11810 -v /home/sdbadmin/data:/data -v /home/sdbadmin/conf/local:/opt/sequoiadb/conf/local registry.cn-hangzhou.aliyuncs.com/508mars/sequoiadb

sdb2:

weave stop

weave launch 192.168.17.132

eval $(weave env)

docker run -dit --name sdbserver2 -p 11810:11810 -v /home/sdbadmin/data:/data -v /home/sdbadmin/conf/local:/opt/sequoiadb/conf/local registry.cn-hangzhou.aliyuncs.com/508mars/sequoiadb

sdb3:

weave stop

weave launch 192.168.17.132

eval $(weave env)

docker run -dit --name sdbserver3 -p 11810:11810 -v /home/sdbadmin/data:/data -v /home/sdbadmin/conf/local:/opt/sequoiadb/conf/local registry.cn-hangzhou.aliyuncs.com/508mars/sequoiadb

192.168.17.132 is the IP address of sdb1 and 11810 is the exposed cluster access port. Volumes that host hosts store node configuration information must be mounted in the container's / opt/sequoiadb/conf/local directory. Volumes that store table data can be mounted in user-defined directories, but once the cluster is created, it cannot be changed. When starting the container, you must specify the machine name, because after building the cluster, the machine name will be stored in the system table of SequoiaDB, and the inconsistency between the machine name of the node and the system table will result in the failure to join the cluster. In the case of weave, it is recommended to use the name option instead of hostname to set the machine name. The latter prevents weave from adding machine names to DNS servers, and weave automatically sets machine names based on the value of name, and adds weave.local domain names to the machine names, and adds them to DNS servers.

STEP6: Copy the script to create the SequoiaDB cluster into the container

d

ocker cp create_cluster.js sdbserver1:/data

 

create_cluster.js The contents are as follows:

var array_hosts = ["sdbserver1.weave.local", "sdbserver2.weave.local", "sdbserver3.weave.local"];

var array_dbroot = ["/data/disk1/sequoiadb/database","/data/disk2/sequoiadb/database","/data/disk3/sequoiadb/database"];

var port_sdbcm = "11790";

var port_temp_coord = "18888";

var cataloggroup = {gname:"SYSCatalogGroup", gport:"11820", ghosts:["sdbserver1.weave.local", "sdbserver2.weave.local", "sdbserver3.weave.local"]};

var array_coordgroups = [

        {gname:"SYSCoord", gport:"11810", ghosts:["sdbserver1.weave.local", "sdbserver2.weave.local", "sdbserver3.weave.local"]}

];

var array_datagroups = [

        {gname:"dg1", gport:"11830", ghosts:["sdbserver1.weave.local", "sdbserver2.weave.local", "sdbserver3.weave.local"], goptions:{transactionon:true}}

       ,{gname:"dg2", gport:"11840", ghosts:["sdbserver1.weave.local", "sdbserver2.weave.local", "sdbserver3.weave.local"], goptions:{transactionon:true}}

       ,{gname:"dg3", gport:"11850", ghosts:["sdbserver1.weave.local", "sdbserver2.weave.local", "sdbserver3.weave.local"], goptions:{transactionon:true}}

];

var array_domains = [

        {dname:"allgroups", dgroups:["dg1", "dg2", "dg3"], doptions:{AutoSplit:true}}

];

 

println("Start Temporary Coordination Node");

var oma = new Oma(array_coordgroups[0].ghosts[0], port_sdbcm);

oma.createCoord(port_temp_coord, array_dbroot[0]+"/coord/"+port_temp_coord);

oma.startNode(port_temp_coord);

 

println("Create catalog node groups:"+cataloggroup.ghosts[0]+"   "+cataloggroup.gport+"    "+array_dbroot[0]+"/cata/"+cataloggroup.gport);

var db = new Sdb(array_coordgroups[0].ghosts[0], port_temp_coord);

db.createCataRG(cataloggroup.ghosts[0], cataloggroup.gport, array_dbroot[0]+"/cata/"+cataloggroup.gport);

var cataRG = db.getRG("SYSCatalogGroup");

for (var i in cataloggroup.ghosts) {

        if (i==0) {continue;}

    println("Create catalog nodes: "+cataloggroup.ghosts[i]+"  "+cataloggroup.gport+"    "+array_dbroot[0]+"/cata/"+cataloggroup.gport);

        var catanode = cataRG.createNode(cataloggroup.ghosts[i], cataloggroup.gport, array_dbroot[0]+"/cata/"+cataloggroup.gport);

        catanode.start();

}

 

println("Creating Coordination Node Group");

var db = new Sdb(array_coordgroups[0].ghosts[0], port_temp_coord);

var coordRG = db.createCoordRG();

for (var i in array_coordgroups) {

        for (var j in array_coordgroups[i].ghosts) {

                println("Creating Coordination Node Group:"+array_coordgroups[i].ghosts[j]+"    "+array_coordgroups[i].gport+"    "+array_dbroot[0]+"/coord/"+array_coordgroups[i].gport);

                coordRG.createNode(array_coordgroups[i].ghosts[j], array_coordgroups[i].gport, array_dbroot[0]+"/coord/"+array_coordgroups[i].gport);

        }

}

coordRG.start();

 

println("Delete temporary coordination nodes")

var oma = new Oma(array_coordgroups[0].ghosts[0], port_sdbcm);

oma.removeCoord(port_temp_coord);

 

println("Create data node groups")

var db = new Sdb(array_coordgroups[0].ghosts[0], array_coordgroups[0].gport);

var k=0;

for (var i in array_datagroups) {

        var dataRG = db.createRG(array_datagroups[i].gname);

        for (var j in array_datagroups[i].ghosts) {

                println("Create data nodes:"+array_datagroups[i].gname+"    "+array_datagroups[i].ghosts[j]+"   "+array_datagroups[i].gport+"    "+array_dbroot[k]+"/data/"+array_datagroups[i].gport+"    "+array_datagroups[i].goptions)

                dataRG.createNode(array_datagroups[i].ghosts[j], array_datagroups[i].gport, array_dbroot[k]+"/data/"+array_datagroups[i].gport, array_datagroups[i].goptions);

        }

        dataRG.start();

        k++;

}

 

println("Create domain");

var db = new Sdb(array_coordgroups[0].ghosts[0], array_coordgroups[0].gport);

for (var i in array_domains) {

        println("Create domain:"+array_domains[i].dname+"   "+array_domains[i].dgroups+"    "+array_domains[i].doptions)

        db.createDomain(array_domains[i].dname, array_domains[i].dgroups, array_domains[i].doptions );

}

STEP7: Creating SequoiaDB Cluster

docker exec sdbserver1 su - sdbadmin -c "sdb -f /data/create_cluster.js"

At this point, the SequoiaDB cluster is created and started, and the cluster will start automatically when the container is restarted later.

1. conclusion

SequoiaDB uses container technology to achieve rapid cluster deployment, which greatly simplifies the difficulty of installation and deployment for beginners. Later, I will do some optimization on the image making of SequoiaDB, because the image is a little big now. The main reason is that using ADD or COPY command to copy the installation medium into the Docker container will generate a new image image1. Although the installation medium is deleted in the final image image2, it is on the image 1, so the size of image 2 still contains the installation medium. . It's better to use ADD copy tar package (ADD will automatically decompress) or similar way:

RUN mkdir -p /usr/src/things \

    && curl -SL http://example.com/big.tar.xz \

    | tar -xJC /usr/src/things \

    && make -C /usr/src/things all