Preface:
In the past, I wrote a tutorial about the creation of docker cluster, but it is not deep enough. It just stays on the initialization of the cluster and the joining of nodes. This time, it is specific to the service and explains the principle in detail. And it summarizes a lot of pits that we have trampled in the process of doing it ourselves (such as the digest problem of mirror image). In the process of consulting information, I also learned that Google's kubernetes and swarm are similar things. I am going to open an entry pit after that. After all, Google's things are very interesting.
Cluster Topology:
Linux version: Red Hat Enterprise Linux Server release 7.3
Doker version:
Client:
Version: 17.03.0-ce
API version: 1.26
Go version: go1.7.5
Git commit: 3a232c8
Built: Tue Feb 28 08:10:07 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.0-ce
API version: 1.26 (minimum version 1.12)
Go version: go1.7.5
Git commit: 3a232c8
Built: Tue Feb 28 08:10:07 2017
OS/Arch: linux/amd64
Experimental: false1. Create docker swarm cluster
Corresponding parts of official documents: Official docker swarm
This one is about the cluster of docker services, not the cluster of services that our container involves.
docker swarm init --advertise-addr 172.25.3.250With this command alone, you can create a docker cluster with 172.25.3.250 as manager, - advertise-addr is not required, but it is specified when you have multiple network cards on your computer.
docker swarm join --token ***** ip:2377Other machines want to join the docker cluster using the above commands.
[root@foundation3 ~]# docker swarm join-token worker
To add a worker to this swarm, run the following command:
docker swarm join \
--token SWMTKN-1-4jlxqqc9jg39a7ll5npktriayhnk040vuunhcwj2h6ntvcy3j0-3x6za4vgrabq5ydivyxvd6kma \
172.25.3.250:2377
[root@foundation3 ~]# docker swarm join-token manager
To add a manager to this swarm, run the following command:
docker swarm join \
--token SWMTKN-1-4jlxqqc9jg39a7ll5npktriayhnk040vuunhcwj2h6ntvcy3j0-602yfdl0qc5zmhcvo1etytu7y \
172.25.3.250:2377
According to the difference of token, we can distinguish whether the manager node joins the cluster or the normal tie node.
[root@foundation3 ~]# docker swarm join-token manager --quiet
SWMTKN-1-4jlxqqc9jg39a7ll5npktriayhnk040vuunhcwj2h6ntvcy3j0-602yfdl0qc5zmhcvo1etytu7y
By adding the quiet parameter, only token can be output, which is useful when token needs to be saved.
2. Deployment of services in docker cluster
1. Attention before deploying services
In accordance with this tutorial, it is recommended that each machine can connect to the external network and download the image from the docker image warehouse. The reason will be explained later.
Speaking of downloading mirrors, many brothers may say that downloading mirrors is too slow. From Docker Hub, here I recommend a domestic mirror accelerator which is very useful. DaoCloud - Innovative Leader in Enterprise Cloud Computing. 
Register your account and enter your own user interface. Click on the circle of my picture in the upper right corner to enter the accelerator interface. 
The accelerator can be configured by directly entering the command indicated. Manual configuration is also possible if the configuration fails using the command he instructs.
[root@foundation3 docker]# pwd
/etc/docker
[root@foundation3 docker]# ls
daemon.json key.json test1.tar test2.tar
[root@foundation3 docker]#
Write the daemon.json file (replace the following * with your private mirror address)
"registry-mirrors":["http://**********"]Now you can have a happy pull image. The mirror we need this time is the mirror of Nginx.
[root@foundation3 docker]# docker search nginx
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
nginx Official build of Nginx. 6085 [OK]
jwilder/nginx-proxy Automated Nginx reverse proxy for docker c... 1038 [OK]
richarvey/nginx-php-fpm Container running Nginx + PHP-FPM capable ... 381 [OK]
jrcs/letsencrypt-nginx-proxy-companion LetsEncrypt container to use with nginx as... 182 [OK]
webdevops/php-nginx Nginx with PHP-FPM 78 [OK]
million12/nginx-php Nginx + PHP-FPM 5.5, 5.6, 7.0 (NG), CentOS... 77 [OK]
From here we can see many images of nginx. According to the number of stars, we can choose the most reliable image. Here we want to emphasize one thing, docker tag. We recommend tagging every image we use, such as my list of Nginx images
[root@foundation3 docker]# docker images | grep nginx
nginx 1.12 17daebd00e2c 2 weeks ago 109 MB
nginx 1.13 3448f27c273f 2 weeks ago 109 MB
These tags are not the default latest, because this default tag can cause a lot of trouble, because many times the image with latest is not necessarily "latest", which can be very disturbing. So now that we use the mirror to take tag, but according to the official docker image information search ed above, there is no tag information, what can we do? I suggest you go to the mirror's official website to check when you need to download the image, but the access speed of DockerHub's official website is too slow. At this time, we still use our domestic docker image source. 
From the interface provided here, we can see Docker Hub's mirror information. 
After entering the details page of nginx, we can see the specific tag. For example, we would like to download version 1.12.0 nginx image using tags such as stable,1.12, etc.
docker pull nginx:1.12
Now that they are all configured, we pull down the images of nginx:1.12 and nginx:1.13 on manager, node1 and node2. If you just pull down on manager, the deployment service will fail.
2. Deployment of single-node services
[root@foundation3 docker]# docker service create --name web_t nginx:1.12
This command lets us deploy a Nginx container on the manager node.
[root@foundation3 docker]# docker service ls
ID NAME MODE REPLICAS IMAGE
gy0slheb99he web_t replicated 1/1 nginx:1.12
Through the REPLICAS project, we can see the start-up status of cluster nodes, "/" on the left is the number of start-ups in the cluster, and "/" on the right is the total number of cluster nodes.
[root@foundation3 docker]# docker service ps web_t
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
6qzlzycuageu web_t.1 nginx:1.12 Nored.example.com Running Running 4 minutes ago Docker services can see which machine the cluster node is running on. Running time, etc. This time it is a single node cluster, we can see that the container runs on the local machine.
3. Deployment of multi-node clusters
[root@foundation3 docker]# docker service create --replicas 3 --name web_a nginx:1.12
Adding replicas 3 means that the number of clusters becomes 3. manager will distribute containers on an average of three nodes
[root@foundation3 docker]# docker service ls
ID NAME MODE REPLICAS IMAGE
q8dsqiesl347 web_a replicated 3/3 nginx:1.12
Through docker services, you can see that all three nodes are running.
[root@foundation3 docker]# docker service ps web_a
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
5pavjoz2f7wi web_a.1 nginx:1.12 worker1.mo.com Running Running 6 minutes ago
ug3hbmlfy95g web_a.2 nginx:1.12 foundation3.ilt.example.com Running Running 6 minutes ago
94dyllk3k1sg web_a.3 nginx:1.12 worker2.mo.com Running Running 6 minutes ago
Through docker service ps, you can see that the nginx container runs on three different hosts. To prove that it runs on another host, we log on to worker1.mo.com.
[root@worker2 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ed905660baeb nginx@sha256:0d71ff22db29a08ac7399d1b35b0311c5b0cbe68d878993718275758811f652a "nginx -g 'daemon ..." 29 minutes ago Up 29 minutes 80/tcp web_a.3.94dyllk3k1sgy44fha1ho0kss
e5423ea1a6aa You can see that the container is running on docker2.
4. Points trampled by the deployment process (explaining why each node has to pull mirror from DockerHub)
When I first deployed the service, because Noe1 and Noe2 could not be connected to the network, I save d the image pull ed down from the Manager directly into the tar package. In Noe1 and Noe2, I used load to import the nginx service deployed after the tar package, but found that the container could only run on the manager, and then the problem recurred.
Disconnect your Noe1 and Noe2 from the external network
[root@worker2 ~]# ls | grep nginx_1.12.tar
nginx_1.12.tar
This is the tar package for the Nginx image, which we imported into Noe1 and Noe2. Before that, delete the original nginx image
docker rmi nginx:1.12Import mirroring
[root@worker2 ~]# docker load -i nginx_1.13.tar
Loaded image: nginx:1.12
f12c15fc56f1: Loading layer [==================================================>] 52.75 MB/52.75 MB
08e6bf75740d: Loading layer [==================================================>] 3.584 kB/3.584 kB
Loaded image ID: sha256:17daebd00e2c95abb02cd48bcbf562c1993460d40d626501f23985126ea26b21
Loaded image ID: sha256:3448f27c273f3122fc554d7acf33796efb4df2ad9886efc092c3bfe716e897b7
Now start redeploying services
[root@foundation3 images]# docker service create --replicas 3 --name web nginx:1.12
x4s7b1iccoebir6avpdnwjcb2
View cluster status
[root@foundation3 images]# docker service ps web
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
gf040hgh3krc web.1 nginx:1.12 foundation3.ilt.example.com Running Preparing 3 seconds ago
jradnn5lgvxl web.2 nginx:1.12 worker2.mo.com Ready Rejected 1 second ago "No such image: nginx@sha256:0…"
tez20wghmtpf \_ web.2 nginx:1.12 worker2.mo.com Shutdown Rejected 3 seconds ago "No such image: nginx@sha256:0…"
yge4dufnmozw web.3 nginx:1.12 worker1.mo.com Ready Rejected 1 second ago "No such image: nginx@sha256:0…"
esprx0otnwj5 \_ web.3 nginx:1.12 worker1.mo.com Shutdown Rejected 2 seconds ago "No such image: nginx@sha256:0…" You can see how Mingming failed to create a load nginx:1.12 image on node1 and node2. According to the display node1 of ps, the reason why node2 refused was that there was no mirror image of nginx. Looking carefully, it was found that there was @sha... We wonder if the inconsistency between the @ value of the imported mirror and the manager leads to the rejection of Noe1 and Noe2.
Now the question arises: how do we determine that nginx:1.12 image is the official image? According to the tag behind it, it is obviously not safe enough. At this time, we need something similar to hash signature to mark, but in docker this value is called digests. Let's look at digests for nginx: 1.12 on manager.
[root@foundation3 images]# docker images --digests|grep nginx
nginx 1.12 sha256:0d71ff22db29a08ac7399d1b35b0311c5b0cbe68d878993718275758811f652a 17daebd00e2c 2 weeks ago 109 MB
Look at the digests of the nginx:1.12 mirror on node2.
[root@worker2 ~]# docker images --digests
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
nginx 1.12 <none> 17daebd00e2c 2 weeks ago 109 MB
You can see that it's a none. Seeing this, I think you should understand why the redeployment service failed because although we can make sure that the nginx mirror is save d and load ed by ourselves, docker can't parse digests according to this tag, so we think that different mirrors lead to the failure of service creation.
Solution: At present, you can only connect all your nodes to the Internet. If your image is not wrong in the save process, then we will connect Noe1 and Noe2 to the external network, then redeploy the cluster Noe1, and the docker on Noe2 will go to DockerHub to query the corresponding digests of tag, and the deployment will be successful. However, in some environments, our nodes can not connect to the external network, how to do, has not yet been resolved, I will update you immediately after the solution. This section deals with knowledge points like digests. Docker official website link.
3. Detailed parameter configuration when creating services and updating deployed services
1. Mount files from the host into the service
[root@foundation3 tmp]# docker service create --name web --mount type=bind,src=/tmp/in_in,dst=/ini nginx:1.12Mount the / tmp/in_in file on the local machine to / ini in the container. This parameter is similar to - v in docker run, but remember to specify the type. 
(Pictures quoted from Docker's website)
Read only mount
[root@foundation3 ~]# docker service create --mount type=bind,src=/tmp/in_in,dst=/in,ro=true --name web nginx:1.12The default mount is read-write mount. If you don't want the container to modify the data, add the ro=true parameter.
2. Update method configuration at creation time
docker service create \
--replicas 12 \
--name my_web \
--update-delay 10s \
--update-parallelism 2 \
--update-failure-action continue \
nginx:1.12The above is the creation of a docker cluster of nodes, which specifies the'- update-delay'to determine the interval between each group of updates, the'- update-parallelism' to determine the number of containers that are updated at the same time, and the'- update-failure-continue'to indicate whether the operation performed when the update fails is allowed to continue or not.
docker service update \ --rollback \ --update-delay 0s web
Above is the command used by the rollback service.