docker service installation
Host System and Doker Version Selection
centos7's support for virtualization is relatively perfect from the point of view of existing systems, and the latest version of docker is of course more functional; therefore, the latest version of Centos7+docker is chosen.
kernel upgrade
Phenomenon description: system crash
Reason Description: centos7.1 default installed core version 3.10, for running a small number of containers, there is no problem, when the number of running containers increases (for example, 100) will find that the operating system will crash after a period of time, resulting in the unavailability of the entire physical machine, after research found that upgrading the system kernel to 4.2 can solve this problem.
Solution:
View the current kernel version of the system
[root@docker6 ~]# uname -a Linux docker6.stg.1qianbao.com 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Upgrade kernel to 4.2
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org yum install http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm yum --enablerepo=elrepo-kernel install kernel-ml grub2-set-default 0
Restart Operating System
reboot
Look at the kernel version again
[root@docker6 ~]# uname -a Linux docker6 4.2.1-1.el7.elrepo.x86_64 #1 SMP Mon Sep 21 20:01:19 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
It can be used if it needs to be unloaded.
yum remove kernel-ml
Install docker
Configure yum source
[root@docker6 ~]# vim /etc/yum.repos.d/docker.repo [docker-epel] name=docker-epel baseurl=http://mirrors.aliyun.com/centos/$releasever/extras/$basearch/ gpgcheck=0
Relevant rpm packages can also be downloaded to the server installation.
Install docker
[root@docker6 ~]# yum install docker -y Loaded plugins: fastestmirror Repository 'moosefs' is missing name in configuration, using id Loading mirror speeds from cached hostfile Resolving Dependencies --> Running transaction check ---> Package docker.x86_64 0:1.8.2-10.el7.centos will be installed --> Processing Dependency: docker-selinux >= 1.8.2-10.el7.centos for package: docker-1.8.2-10.el7.centos.x86_64 --> Running transaction check ---> Package docker-selinux.x86_64 0:1.8.2-10.el7.centos will be installed --> Finished Dependency Resolution Dependencies Resolved =============================================================================================================================================================================================== Package Arch Version Repository Size =============================================================================================================================================================================================== Installing: docker x86_64 1.8.2-10.el7.centos extras 10 M Installing for dependencies: docker-selinux x86_64 1.8.2-10.el7.centos extras 63 k Transaction Summary =============================================================================================================================================================================================== Install 1 Package (+1 Dependent package) Total download size: 10 M Installed size: 46 M Downloading packages: (1/2): docker-selinux-1.8.2-10.el7.centos.x86_64.rpm | 63 kB 00:00:00 (2/2): docker-1.8.2-10.el7.centos.x86_64.rpm | 10 MB 00:00:00 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 39 MB/s | 10 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : docker-selinux-1.8.2-10.el7.centos.x86_64 1/2 Installing : docker-1.8.2-10.el7.centos.x86_64 2/2 warning: /etc/sysconfig/docker created as /etc/sysconfig/docker.rpmnew Verifying : docker-selinux-1.8.2-10.el7.centos.x86_64 1/2 Verifying : docker-1.8.2-10.el7.centos.x86_64 2/2 Installed: docker.x86_64 0:1.8.2-10.el7.centos Dependency Installed: docker-selinux.x86_64 0:1.8.2-10.el7.centos Complete!
Start docker service
[root@docker6 ~]# systemctl start docker.service
View the docker version
[root@docker6 ~]# docker -v Docker version 1.7.1, build 786b29d/1.7.1
Draw docker image
Customized warehousing is used here. When using, please modify it according to the actual situation.
Search for mirrors in the warehouse
If the warehouse is not an https protocol, it needs to be used in the docker service parameter -- insecure-registry=xxoo.com to add to the trust list and restart the service
[root@docker6 ~]# docker search 1qianbao.com/jboss INDEX NAME DESCRIPTION STARS OFFICIAL AUTOMATED 1qianbao.com 1qianbao.com/library/centos6.5_jboss 0 1qianbao.com 1qianbao.com/library/centos6.5_jboss_base 0 1qianbao.com 1qianbao.com/library/jboss 0 1qianbao.com 1qianbao.com/library/jboss_logstash 0
Draw it to the local area
[root@docker6 ~]# docker pull 1qianbao.com/jboss Using default tag: latest 512b1dcc52f6: Download complete 0757207a8bea: Download complete bf839298b71b: Download complete 15661f85a6fd: Download complete d4f8087e2cbc: Download complete 7aebe5484289: Download complete 35bd70dfe627: Download complete 070eee356b7c: Download complete 6559d408744a: Download complete d9406fd1f731: Download complete Status: Downloaded newer image for 1qianbao.com/jboss:latest
View local mirrors
[root@docker6 ~]# docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE 1qianbao.com/jboss latest 512b1dcc52f6 7 weeks ago 832.8 MB
Create a container
[root@docker6 ~]# docker run -id -h test_hostname --name test_conftainer_name 1qianbao.com/jboss /bin/bash 2744cb820ec0b2d8a80d347df3d6631703822841066954ad1dbd3ba779fb09c2 [root@docker6 ~]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2744cb820ec0 1qianbao.com/jboss "/bin/bash" 59 seconds ago Up 58 seconds 22/tcp test_conftainer_name [root@docker6 ~]# docker exec test_conftainer_name whoami root
Customize service startup parameters
Modify configuration files
[root@docker4 ~]# cat /etc/sysconfig/docker # /etc/sysconfig/docker # Modify these options if you want to change the way the docker daemon runs OPTIONS='--selinux-enabled=false --storage-driver=devicemapper --storage-opt dm.basesize=100G --insecure-registry=1qianbao.com -b br0 -H unix:///var/run/docker.sock -H tcp://0.0.0.0:9999'
Parameter detailed explanation
--storage-driver=devicemapper #Driven by device mapper mode, the driving mode can be described in detail with reference to [http://blog.csdn.net/qq_26923057/article/details/52351731][1] --storage-opt dm.basesize=100G #Allocate 100G disk space for each container --selinux-enabled=false #Disable selinux in container --insecure-registry=1qianbao.com #Specify the storage address for http protocol access -H unix:/// var/run/docker.sock # Specifies the socket file location of the docker service, which can be accessed by other applications via socket -H tcp://0.0.0:9999 # Specify docker service remote api port, accessible via docker-py remote connection -b br0 #Container network is bridged to local br0 network card
After the modification, restart the docker service
Service tuning
Kernel tuning
Phenomenon Description: Network High Delay, Packet Loss
Reason description: When a single host runs more than 100 containers, the pressure on the host increases correspondingly, especially in the network aspect.
Take ARP as an example: Containers use br0 network to communicate with the outside world, so each container will cache the ARP information. If 100 containers store the gateway and host information into the ARP caching table respectively, there will be 200 ARP caching tables on the physical machine. The default ARP caching entry of the system is 128, so a part of the ARP information will be covered. If there is frequent network communication between containers, there will be a lot of delay and even packet loss.
[root@docker6 ~]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh1 #System default ARP cache entry 128 [root@docker6 ~]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh2 #Soft Limitation of System ARP Item Value 512 [root@docker6 ~]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh3 #Hard Limitation of System ARP Item Value 1024
Solution:
Modifying System Kernel Parameters
[root@docker6 ~]# vim /etc/sysctl.conf # System default settings live in /usr/lib/sysctl.d/00-system.conf. # To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file # # For more information, see sysctl.conf(5) and sysctl.d(5). net.ipv4.ip_forward=1 net.ipv4.neigh.default.gc_thresh1=25600 net.ipv4.neigh.default.gc_thresh2=51200 net.ipv4.neigh.default.gc_thresh3=102400 net.core.rmem_default = 2097152 net.core.rmem_max=16777216 net.core.wmem_max=16777216 net.core.wmem_default = 2097152 net.ipv4.neigh.default.base_reachable_time = 172800 net.ipv4.neigh.default.gc_stale_time = 172800 kernel.msgmax=65536 kernel.msgmnb=65536
Restart or execute the following order to take effect
[root@docker6 ~]# sysctl -p net.ipv4.ip_forward = 1 net.ipv4.neigh.default.gc_thresh1 = 25600 net.ipv4.neigh.default.gc_thresh2 = 51200 net.ipv4.neigh.default.gc_thresh3 = 102400 net.core.rmem_default = 2097152 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.core.wmem_default = 2097152 sysctl: net.ipv4.neigh.default.base_reachable_time is deprecated, value not set net.ipv4.neigh.default.gc_stale_time = 172800 kernel.msgmax = 65536 kernel.msgmnb = 65536
ulimit tuning
Phenomenon Description: Resource Depletion When Containers Start Services
Cause Description: In the case of large host, when multiple containers start multi-process or multi-threaded services in parallel, a large amount of file handles are consumed, resulting in resource exhaustion.
Solution: Modify the nproc limitation of the system for ordinary users
[root@docker6 ~]# cat /etc/security/limits.d/20-nproc.conf # Default limit for number of user's processes to prevent # accidental fork bombs. # See rhbz #432903 for reasoning. #* soft nproc 4096 root soft nproc unlimited
Detecting the maximum number of open handles on the host can be used: for I in {1.200000}; do sleep 2000 & done;
data resource pool tuning
By default, a 107.4G data file is created when the docker service starts, and all changes to the booted container are stored in the data file; that is to say, when the relevant data data generated in the container exceeds 107.4G, the container will no longer have extra space available, resulting in the root directory of all the containers becoming read-only!
The docker info information for the host is as follows
[root@docker6 ~]# docker info Containers: 1 Images: 10 Storage Driver: devicemapper Pool Name: docker-253:0-4298463412-pool Pool Blocksize: 65.54 kB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 2.749 GB Data Space Total: 107.4 GB Data Space Available: 104.6 GB Metadata Space Used: 2.437 MB Metadata Space Total: 2.147 GB Metadata Space Available: 2.145 GB Udev Sync Supported: true Deferred Removal Enabled: false Data loop file: /var/lib/docker/devicemapper/devicemapper/data Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.107-RHEL7 (2015-10-14) Execution Driver: native-0.2 Logging Driver: json-file Kernel Version: 3.10.0-327.el7.x86_64 Operating System: CentOS Linux 7 (Core) CPUs: 48 Total Memory: 251.8 GiB Name: docker6.stg.1qianbao.com ID: 6YYO:CCO3:LBQN:7MDT:K4XI:JFSP:THO2:KQP2:TJJ5:EVHU:TZLC:W2BZ WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled
Solution: Replace data pool files with disk partitions, lvs logical volumes, and virtual large files
In this example, the data pool file is replaced by a virtual large file. The steps are as follows:
[root@docker6 ~]# cat init_docker.sh #!/bin/bash echo "stop docker service" service docker stop echo "delete the docker data pool" /bin/rm -rf /var/lib/docker/ echo "create docker data pool" mkdir -p /var/lib/docker/devicemapper/devicemapper/ dd if=/dev/zero bs=1G count=0 of=/var/lib/docker/devicemapper/devicemapper/data seek=1500 dd if=/dev/zero bs=1G count=0 of=/var/lib/docker/devicemapper/devicemapper/metadata seek=2 #clear veth net ifconfig | grep veth | awk -F ':' '{print $1}' | while read LINE;do ip link delete $LINE;done echo "start docker service" service docker start [root@docker6 ~]# bash init_docker.sh stop docker service Redirecting to /bin/systemctl stop docker.service delete the docker data pool create docker data pool 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000381916 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000156311 s, 0.0 kB/s start docker service Redirecting to /bin/systemctl start docker.service
After executing the above command, docker info information is as follows
[root@docker6 ~]# docker info Containers: 0 Images: 0 Storage Driver: devicemapper Pool Name: docker-253:0-6501508-pool Pool Blocksize: 65.54 kB Backing Filesystem: xfs Data file: /dev/loop2 Metadata file: /dev/loop3 Data Space Used: 1.821 GB Data Space Total: 1.611 TB Data Space Available: 1.491 TB Metadata Space Used: 7.307 MB Metadata Space Total: 2.147 GB Metadata Space Available: 2.14 GB Udev Sync Supported: true Deferred Removal Enabled: false Data loop file: /var/lib/docker/devicemapper/devicemapper/data Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.107-RHEL7 (2015-10-14) Execution Driver: native-0.2 Logging Driver: json-file Kernel Version: 3.10.0-327.el7.x86_64 Operating System: CentOS Linux 7 (Core) CPUs: 48 Total Memory: 251.8 GiB Name: docker6.stg.1qianbao.com ID: 6YYO:CCO3:LBQN:7MDT:K4XI:JFSP:THO2:KQP2:TJJ5:EVHU:TZLC:W2BZ WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled