Docker Container Technology Warfare

Keywords: Linux Docker CentOS SELinux

docker service installation

Host System and Doker Version Selection

centos7's support for virtualization is relatively perfect from the point of view of existing systems, and the latest version of docker is of course more functional; therefore, the latest version of Centos7+docker is chosen.

kernel upgrade

Phenomenon description: system crash
Reason Description: centos7.1 default installed core version 3.10, for running a small number of containers, there is no problem, when the number of running containers increases (for example, 100) will find that the operating system will crash after a period of time, resulting in the unavailability of the entire physical machine, after research found that upgrading the system kernel to 4.2 can solve this problem.
Solution:
View the current kernel version of the system

[root@docker6 ~]# uname -a
Linux docker6.stg.1qianbao.com 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Upgrade kernel to 4.2

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
yum install http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install kernel-ml
grub2-set-default 0

Restart Operating System

reboot

Look at the kernel version again

[root@docker6 ~]# uname -a
Linux docker6 4.2.1-1.el7.elrepo.x86_64 #1 SMP Mon Sep 21 20:01:19 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

It can be used if it needs to be unloaded.

yum remove kernel-ml

Install docker

Configure yum source

[root@docker6 ~]# vim /etc/yum.repos.d/docker.repo
[docker-epel]
name=docker-epel
baseurl=http://mirrors.aliyun.com/centos/$releasever/extras/$basearch/
gpgcheck=0

Relevant rpm packages can also be downloaded to the server installation.

Install docker

[root@docker6 ~]# yum install docker -y
Loaded plugins: fastestmirror
Repository 'moosefs' is missing name in configuration, using id
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package docker.x86_64 0:1.8.2-10.el7.centos will be installed
--> Processing Dependency: docker-selinux >= 1.8.2-10.el7.centos for package: docker-1.8.2-10.el7.centos.x86_64
--> Running transaction check
---> Package docker-selinux.x86_64 0:1.8.2-10.el7.centos will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===============================================================================================================================================================================================
 Package                                         Arch                                    Version                                                 Repository                               Size
===============================================================================================================================================================================================
Installing:
 docker                                          x86_64                                  1.8.2-10.el7.centos                                     extras                                   10 M
Installing for dependencies:
 docker-selinux                                  x86_64                                  1.8.2-10.el7.centos                                     extras                                   63 k

Transaction Summary
===============================================================================================================================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 10 M
Installed size: 46 M
Downloading packages:
(1/2): docker-selinux-1.8.2-10.el7.centos.x86_64.rpm                                                                                                                    |  63 kB  00:00:00     
(2/2): docker-1.8.2-10.el7.centos.x86_64.rpm                                                                                                                            |  10 MB  00:00:00     
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                           39 MB/s |  10 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : docker-selinux-1.8.2-10.el7.centos.x86_64                                                                                                                                   1/2 
  Installing : docker-1.8.2-10.el7.centos.x86_64                                                                                                                                           2/2 
warning: /etc/sysconfig/docker created as /etc/sysconfig/docker.rpmnew
  Verifying  : docker-selinux-1.8.2-10.el7.centos.x86_64                                                                                                                                   1/2 
  Verifying  : docker-1.8.2-10.el7.centos.x86_64                                                                                                                                           2/2 

Installed:
  docker.x86_64 0:1.8.2-10.el7.centos                                                                                                                                                          

Dependency Installed:
  docker-selinux.x86_64 0:1.8.2-10.el7.centos                                                                                                                                                  

Complete!

Start docker service

[root@docker6 ~]# systemctl start docker.service

View the docker version

[root@docker6 ~]# docker -v
Docker version 1.7.1, build 786b29d/1.7.1

Draw docker image

Customized warehousing is used here. When using, please modify it according to the actual situation.

Search for mirrors in the warehouse

If the warehouse is not an https protocol, it needs to be used in the docker service parameter -- insecure-registry=xxoo.com to add to the trust list and restart the service

[root@docker6 ~]# docker search 1qianbao.com/jboss
INDEX          NAME                                        DESCRIPTION   STARS     OFFICIAL   AUTOMATED
1qianbao.com   1qianbao.com/library/centos6.5_jboss                      0                    
1qianbao.com   1qianbao.com/library/centos6.5_jboss_base                 0                    
1qianbao.com   1qianbao.com/library/jboss                                0                    
1qianbao.com   1qianbao.com/library/jboss_logstash                       0 

Draw it to the local area

[root@docker6 ~]# docker pull 1qianbao.com/jboss
Using default tag: latest
512b1dcc52f6: Download complete 
0757207a8bea: Download complete 
bf839298b71b: Download complete 
15661f85a6fd: Download complete 
d4f8087e2cbc: Download complete 
7aebe5484289: Download complete 
35bd70dfe627: Download complete 
070eee356b7c: Download complete 
6559d408744a: Download complete 
d9406fd1f731: Download complete 
Status: Downloaded newer image for 1qianbao.com/jboss:latest

View local mirrors

[root@docker6 ~]# docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
1qianbao.com/jboss   latest              512b1dcc52f6        7 weeks ago         832.8 MB

Create a container

[root@docker6 ~]# docker run -id -h test_hostname --name test_conftainer_name 1qianbao.com/jboss /bin/bash
2744cb820ec0b2d8a80d347df3d6631703822841066954ad1dbd3ba779fb09c2
[root@docker6 ~]# docker ps -a
CONTAINER ID        IMAGE                COMMAND             CREATED             STATUS              PORTS               NAMES
2744cb820ec0        1qianbao.com/jboss   "/bin/bash"         59 seconds ago      Up 58 seconds       22/tcp              test_conftainer_name
[root@docker6 ~]# docker exec test_conftainer_name whoami
root

Customize service startup parameters

Modify configuration files

[root@docker4 ~]# cat /etc/sysconfig/docker
# /etc/sysconfig/docker
# Modify these options if you want to change the way the docker daemon runs
OPTIONS='--selinux-enabled=false --storage-driver=devicemapper --storage-opt dm.basesize=100G --insecure-registry=1qianbao.com -b br0 -H unix:///var/run/docker.sock -H tcp://0.0.0.0:9999'

Parameter detailed explanation

--storage-driver=devicemapper            #Driven by device mapper mode, the driving mode can be described in detail with reference to [http://blog.csdn.net/qq_26923057/article/details/52351731][1]
--storage-opt dm.basesize=100G           #Allocate 100G disk space for each container
--selinux-enabled=false                  #Disable selinux in container
--insecure-registry=1qianbao.com         #Specify the storage address for http protocol access
-H unix:/// var/run/docker.sock # Specifies the socket file location of the docker service, which can be accessed by other applications via socket
-H tcp://0.0.0:9999 # Specify docker service remote api port, accessible via docker-py remote connection
-b br0                                   #Container network is bridged to local br0 network card

After the modification, restart the docker service

Service tuning

Kernel tuning

Phenomenon Description: Network High Delay, Packet Loss
Reason description: When a single host runs more than 100 containers, the pressure on the host increases correspondingly, especially in the network aspect.
Take ARP as an example: Containers use br0 network to communicate with the outside world, so each container will cache the ARP information. If 100 containers store the gateway and host information into the ARP caching table respectively, there will be 200 ARP caching tables on the physical machine. The default ARP caching entry of the system is 128, so a part of the ARP information will be covered. If there is frequent network communication between containers, there will be a lot of delay and even packet loss.

[root@docker6 ~]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh1    #System default ARP cache entry
128
[root@docker6 ~]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh2    #Soft Limitation of System ARP Item Value
512
[root@docker6 ~]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh3    #Hard Limitation of System ARP Item Value
1024

Solution:
Modifying System Kernel Parameters

[root@docker6 ~]# vim /etc/sysctl.conf
# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

net.ipv4.ip_forward=1
net.ipv4.neigh.default.gc_thresh1=25600
net.ipv4.neigh.default.gc_thresh2=51200
net.ipv4.neigh.default.gc_thresh3=102400
net.core.rmem_default = 2097152
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.wmem_default = 2097152
net.ipv4.neigh.default.base_reachable_time = 172800
net.ipv4.neigh.default.gc_stale_time = 172800
kernel.msgmax=65536
kernel.msgmnb=65536

Restart or execute the following order to take effect

[root@docker6 ~]# sysctl -p
net.ipv4.ip_forward = 1
net.ipv4.neigh.default.gc_thresh1 = 25600
net.ipv4.neigh.default.gc_thresh2 = 51200
net.ipv4.neigh.default.gc_thresh3 = 102400
net.core.rmem_default = 2097152
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.wmem_default = 2097152
sysctl: net.ipv4.neigh.default.base_reachable_time is deprecated, value not set
net.ipv4.neigh.default.gc_stale_time = 172800
kernel.msgmax = 65536
kernel.msgmnb = 65536

ulimit tuning

Phenomenon Description: Resource Depletion When Containers Start Services
Cause Description: In the case of large host, when multiple containers start multi-process or multi-threaded services in parallel, a large amount of file handles are consumed, resulting in resource exhaustion.
Solution: Modify the nproc limitation of the system for ordinary users

[root@docker6 ~]# cat /etc/security/limits.d/20-nproc.conf 
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

#*          soft    nproc     4096
root       soft    nproc     unlimited

Detecting the maximum number of open handles on the host can be used: for I in {1.200000}; do sleep 2000 & done;

data resource pool tuning

By default, a 107.4G data file is created when the docker service starts, and all changes to the booted container are stored in the data file; that is to say, when the relevant data data generated in the container exceeds 107.4G, the container will no longer have extra space available, resulting in the root directory of all the containers becoming read-only!

The docker info information for the host is as follows

[root@docker6 ~]# docker info
Containers: 1
Images: 10
Storage Driver: devicemapper
 Pool Name: docker-253:0-4298463412-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.749 GB
 Data Space Total: 107.4 GB
 Data Space Available: 104.6 GB
 Metadata Space Used: 2.437 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.145 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2015-10-14)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 48
Total Memory: 251.8 GiB
Name: docker6.stg.1qianbao.com
ID: 6YYO:CCO3:LBQN:7MDT:K4XI:JFSP:THO2:KQP2:TJJ5:EVHU:TZLC:W2BZ
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Solution: Replace data pool files with disk partitions, lvs logical volumes, and virtual large files
In this example, the data pool file is replaced by a virtual large file. The steps are as follows:

[root@docker6 ~]# cat init_docker.sh 
#!/bin/bash

echo "stop docker service"
service docker stop

echo "delete the docker data pool"
/bin/rm -rf /var/lib/docker/

echo "create docker data pool"
mkdir -p /var/lib/docker/devicemapper/devicemapper/
dd if=/dev/zero bs=1G count=0 of=/var/lib/docker/devicemapper/devicemapper/data seek=1500
dd if=/dev/zero bs=1G count=0 of=/var/lib/docker/devicemapper/devicemapper/metadata seek=2

#clear veth net
ifconfig | grep veth | awk -F ':' '{print $1}' | while read LINE;do ip link delete $LINE;done

echo "start docker service"
service docker start

[root@docker6 ~]# bash init_docker.sh 
stop docker service
Redirecting to /bin/systemctl stop  docker.service
delete the docker data pool
create docker data pool
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000381916 s, 0.0 kB/s
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000156311 s, 0.0 kB/s
start docker service
Redirecting to /bin/systemctl start  docker.service

After executing the above command, docker info information is as follows

[root@docker6 ~]# docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker-253:0-6501508-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: /dev/loop2
 Metadata file: /dev/loop3
 Data Space Used: 1.821 GB
 Data Space Total: 1.611 TB
 Data Space Available: 1.491 TB
 Metadata Space Used: 7.307 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.14 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2015-10-14)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 48
Total Memory: 251.8 GiB
Name: docker6.stg.1qianbao.com
ID: 6YYO:CCO3:LBQN:7MDT:K4XI:JFSP:THO2:KQP2:TJJ5:EVHU:TZLC:W2BZ
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Making Mirror Documents

Posted by e-novative on Sun, 14 Apr 2019 15:21:31 -0700