1 What is a Docker image
The Docker image is an overlay of the file system. At the bottom is a file boot system, bootfs. Docker users will not have direct interaction with the boot file system. The second layer of the Docker image is the root file system rootfs, which is usually one or more operating systems, such as ubuntu. In Docker, the file system is always read-only, and each time it is modified, it is copied over to form the final file system. Docker refers to such files as mirrors. One mirror can iterate over the top of the other. The mirror located below is called the parent image, and the bottom image is called the base image. Finally, when a container is launched from a mirror, Docker loads a read-write file system at the top level as the container.
Docker's mechanism is what we call write-time replication.
When a container is deleted, its own Writable layer is deleted together.
Contents of the 2 Docker mirror
Given an overview of what a Docker image is, let's see what it contains.
- The Docker image represents the file system content of a container;
- Initial contact with the federated file system. The concept of a federated file system is a mirror-level management technique in which each layer of mirroring is part of the contents of a container file system.
- Containers are a dynamic environment. Files in each layer of the image are static content. However, ENV, VOLUME, CMD and other contents in the Dockerfile eventually need to be implemented into the running environment of the container. These contents cannot be directly located in the file system content contained in each layer of the image. Then each Docker image will also contain the relationship between the json file record and the container.
3 Mirror Warehouse Service
Docker mirrors are stored in the Image Registry
The Docker client's mirror warehouse service is configurable and uses Docker Hub by default
The mirror warehouse service contains multiple image repositories. Similarly, a mirror warehouse can contain multiple mirrors
The following diagram shows a mirror warehouse service with three mirror warehouses, each of which contains one or more mirrors
4 docker storage driver
docker provides a variety of storage drivers to store mirrors in different ways. Here are several common storage drivers
AUFS (AnotherUnionFS) is a Union FS and is a file-level storage driver. AUFS is a layered file system that transparently covers one or more existing file systems, combining layers into a single representation of the file system. Simply put, it supports mounting different directories to a file system under the same virtual file system. This file system can overlay modified files one by one. No matter how many layers are read-only below, only the top file system is writable. When a file needs to be modified, AUFS creates a copy of the file, uses COW (copy-on-write) to copy the file from the read-only layer to the writable layer for modification, and saves the results in the writable layer. In Docker, the bottom read-only layer is image, and the writable layer is Container. Docker on ubuntu uses AUFS by default.
Overlay is supported after the Linux kernel 3.18 and is also a Union FS. Unlike AUFS, there are only two layers of Overlay: an upperfile system and a lower file system, representing the mirror and container layers of Docker, respectively. When a file needs to be modified, COW is used to copy the file from a read-only lower to a writable upper for modification, and the results are saved in the upper layer. In Docker, the bottom read-only layer is image, and the writable layer is Container. The latest OverlayFS is Overlay2.
AUFS and Overlay are both federated file systems, but AUFS has multiple layers, while Overlay has only two, so AUSF is slower when copying large files with lower layers when writing. And Overlay incorporates the linux kernel mainline, which AUFS does not. At present, AUFS has been basically eliminated.
4.3 DeviceMapper (Device Mapper)
Device mapper is supported after Linux kernel 2.6.9. It provides a mapping framework mechanism from logical devices to physical devices, under which users can easily formulate management strategies to implement storage resources according to their needs. AUFS and OverlayFS are file-level storage, while Device mapper is block-level storage, and all operations operate directly on blocks, not files. The Device mapper driver first creates a resource pool on the block device, then creates a basic device with a file system on the resource pool. All mirrors are snapshots of this basic device, while containers are snapshots of mirrors. So you see in the container that the file system is a snapshot of the file system of the basic device on the resource pool, and no space is allocated to the container. When a new file is to be written, a new block is allocated to it within the mirror of the container and data is written. This is called time-of-use allocation. When you want to modify an existing file, use CoW to allocate block space for the container snapshot, and copy the modified data into a new block in the container snapshot before making the modification.
OverlayFS is file-level storage, Device mapper is block-level storage. When a file is very large and the content modified is small, Overlay copies the entire file regardless of the size of the content modified. It obviously takes more time to modify a large file than a small one. Block-level copies only the blocks that need to be modified, not the entire file. In this scenario, apparently the device mapper is faster. Because the block level has direct access to the logical disk, it is suitable for IO-intensive scenarios. Overlay is relatively better for scenarios with complex internal programs, large concurrencies, but less IO.
BTRFS (often referred to as Butter FS), a COW(copy-on-write) file system announced and in progress by Oracle in 2007. The goal is to replace Linux's ext3 file system, improve ext3 limitations, especially for single file size, total file system size limitations, and add file checking and features. Add features not supported by ext3/4, such as snapshots for writable disks, snapshots for recursive snapshots, RAID support, and subvolume concepts to allow online resizing of file systems.
The purpose of VFS (virtual File System) is to use standard Unix system calls to read and write different file systems on different physical media, which provides a unified operation interface and application programming interface for various file systems. VFS is a glue layer that allows system calls such as open(), read(), write() to work without concern for underlying storage media and file system types.
5 Docker registry
When the container is started, docker daemon attempts to get the associated image locally, which is downloaded from Registry and saved locally when the local image does not exist.
Registry is used to save docker images, including the hierarchy and metadata of the images. Users can build their own Registry or use the official Docker Hub.
Classification of docker registry
- Sponsor Registry: Third-party Registry for use by customers and the Docker community
- Mirror Registry: Third-party Registry, for customer use only
- Vendor Registry: Regisry provided by vendor publishing docker image
- Private Registry: registry provided through a private entity with a firewall and additional security layer
Composition of docker registry
- A mirror repository consisting of all iterated versions of a particular docker image
- Multiple Repositories may exist in a Registry
- Repository can be divided into Top Warehouse and User Warehouse
- User repository name format is User/Repository Name
- Each warehouse can contain multiple Tags, each with a mirror image
- Maintain information about user accounts, mirror checks, and public namespaces
- Equivalent to providing Registry with a retrieval interface that performs functions such as user authentication
6 Docker Hub
Docker Hub is a cloud-based registration service that allows you to link to code libraries, build and test your mirrors, store manually pushed mirrors, and link to Docker Cloud so you can deploy your mirrors on your host.
It provides centralized resources for container image discovery, distribution and change management, user and team collaboration, and workflow automation throughout the development pipeline.
Docker Hub provides the following main features
- Mirror Repository
- Find and extract mirrors from community and official libraries, and manage, push, and extract images from private image libraries that you can access.
- Automated Build
- Automatically create a new image when you change the source code repository
- A webhook is a feature of automated build that allows you to trigger actions after a successful push to the repository.
- Organization Management
- Create a workgroup to manage access to the image repository
- GitHub and Bitbucket Integration
- Add hub and Docker images to the current workflow
7 Docker Image Creation and Generation
In most cases, we do mirroring based on an existing basic image of someone else, which we call a base image. For example, a clean version of the minimal centos, ubuntu, or debian
7.1 Mirror Generation
Ways to Generate Mirrors
- Container-based manufacturing
- Docker Hub automated builds
7.2 Container-based mirroring
Create a new image based on container changes
Grammar: docker commit [OPTIONS] CONTAINER [REPOSITORY[:TAG]]
|—author, -a||Designated Author|
|-c, --change list||Apply the Dockerfile directive to the created image|
|-m, --message string||Submit message|
|-p, --pause||true||Suspend container during submission|
7.2.1 Docker Image Acquisition
To get a Docker image from a remote Registry (such as your own Docker Registry) and add it to your local system, you need to use the docker pull command. I use here to pull a mirror from Docker hub
Grammar: docker pull <registry>[:<port>]/[<namespace>/]<name>:<tag> [root@Docker ~]# docker pull centos Using default tag: latest latest: Pulling from library/centos a1d0c7532777: Pull complete Digest: sha256:a27fd8080b517143cbbbab9dfb7c8571c40d67d534bbdee55bd6c473f432b177 Status: Downloaded newer image for centos:latest docker.io/library/centos:latest
7.2.2 Create and run containers
//Run the container, --name specifies the name of the container, tty type, default is sh [root@Docker ~]# docker run --name nginx -itd 5d0da3dc9764 /bin/bash 95f278ee0a241d28280f967c3878ee45d40b56f50d2a7bab3d2f25e511ed8b39
7.2.3 Enter the container and compile and install nginx
use first docker cp take nginx Packages are passed to containers [root@Docker ~]# docker cp /usr/src/nginx-1.20.2.tar.gz 95f278ee0a24:/usr/src/ [root@Docker ~]# docker exec -it 95f278ee0a24 /bin/bash [root@95f278ee0a24 /]# ls /usr/src/ debug kernels nginx-1.20.2.tar.gz //Create system user nginx [root@95f278ee0a24 /]# useradd -r -m -s /sbin/nologin nginx [root@95f278ee0a24 /]# id nginx uid=998(nginx) gid=996(nginx) groups=996(nginx) //Install dependent packages and compilation tools [root@95f278ee0a24 /]# yum -y install pcre-devel openssl openssl-devel gd-devel gcc gcc-c++ make [root@95f278ee0a24 /]# yum -y groups mark install 'Development Tools' //Create log store directory [root@95f278ee0a24 /]# mkdir -p /var/log/nginx [root@95f278ee0a24 /]# chown -R nginx.nginx /var/log/nginx/ //Compile and Install nginx [root@95f278ee0a24 ~]# cd /usr/src/ [root@95f278ee0a24 src]# ls debug kernels nginx-1.20.2.tar.gz [root@95f278ee0a24 src]# tar xf nginx-1.20.2.tar.gz [root@95f278ee0a24 src]# cd nginx-1.20.2 [root@95f278ee0a24 nginx-1.20.2]# ./configure \ --prefix=/usr/local/nginx \ --user=nginx \ --group=nginx \ --with-debug \ --with-http_ssl_module \ --with-http_realip_module \ --with-http_image_filter_module \ --with-http_gunzip_module \ --with-http_gzip_static_module \ --with-http_stub_status_module \ --http-log-path=ar/loginx/access.log \ --error-log-path=ar/loginx/error.log [root@95f278ee0a24 nginx-1.20.2]# make && make install [root@95f278ee0a24 ~]# ls /usr/local/ bin etc games include lib lib64 libexec nginx sbin share src //Configuring environment variables [root@95f278ee0a24 ~]# echo 'export PATH=/usr/local/nginx/sbin:$PATH' > /etc/profile.d/nginx.sh [root@95f278ee0a24 ~]# source /etc/profile.d/nginx.sh //start nginx [root@95f278ee0a24 ~]# nginx [root@95f278ee0a24 ~]# ss -antl State Recv-Q Send-Q Local Address:Port Peer Address:Port Process LISTEN 0 128 0.0.0.0:80 0.0.0.0:* //Create web test content [root@6476d875f2b4 ~]# echo Nginx to work > /usr/local/nginx/html/index.html [root@6476d875f2b4 ~]# cat /usr/local/nginx/html/index.html Nginx to work //Access on Host [root@Docker ~]# curl 172.17.0.2 Nginx to work //This is because the ports in the docker container do not map all that are inaccessible on the browser
7.2.4 Creating a Nginx image
When creating a mirror, we cannot close the container, we must keep it running, so we must start a new terminal and execute it
The default is to start the sh process, but we want to start a process that you roll down, so we set the container's default start process to nginx when creating the image, so that we can quickly build a simple nginx service with the newly generated image
[root@Docker ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9408c7af4b18 286644a4d832 "/bin/bash" 13 seconds ago Up 12 seconds nginx [root@Docker ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE centos latest 5d0da3dc9764 2 months ago 231MB [root@Docker ~]# docker commit -a 'email@example.com' -c 'CMD ["/usr/local/nginx/sbin/nginx","-g","daemon off;"]' -p 9408c7af4b18 zhaojie10/nginx:v1.20.2 sha256:6f5b02e61ad49adeae10eabab8aa49082c8721eeb731288d0d24d7aa96a5d823 [root@Docker ~]# [root@Docker ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE zhaojie10/nginx v1.20.2 6f5b02e61ad4 6 seconds ago 549MB centos latest 5d0da3dc9764 2 months ago 231MB
7.2.5 Create a new container with the mirror you just created and set the port mapping
[root@Docker ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE zhaojie10/nginx v1.20.2 6f5b02e61ad4 About a minute ago 549MB centos latest 5d0da3dc9764 2 months ago 231MB [root@Docker ~]# docker run -dit --name nginx -p 80:80 6f5b02e61ad4 d9526158bdc16fb347d6c5a86a10f119ca138aa65c72798c77ac8d0984f36057 [root@Docker ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d9526158bdc1 6f5b02e61ad4 "/usr/local/nginx/sb..." 4 seconds ago Up 2 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp nginx [root@Docker ~]# docker exec -it d9526158bdc1 /bin/bash [root@d9526158bdc1 /]# ss -antl State Recv-Q Send-Q Local Address:Port Peer Address:Port Process LISTEN 0 128 0.0.0.0:80 0.0.0.0:* //The container starts the nginx service at this time
Now you can access it on your browser
7.3 Upload to docker hub
Note at this point that our mirror name is zhaojie10/nginx, so we'll create a warehouse named zhaojie10/nginx on Docker Hub, and then push our mirror up
//Log in to docker hub to upload a mirror [root@Docker ~]# docker login Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one. Username: zhaojie10 Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded [root@Docker ~]# docker push zhaojie10/nginx:v2.0 The push refers to repository [docker.io/zhaojie10/nginx] 54bfe1169208: Pushed 27c919eb23ea: Pushed 74ddd0ec08fa: Mounted from library/centos v2.0: digest: sha256:c62e7b10c8c6fcefa9d7248d466b278abc70cc3be9dc9a1a730400d2702f06b6 size: 950
//Pull the container you just uploaded locally to verify that the operation was successful [root@Docker ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE zhaojie10/nginx v1.0 81a4a6c1034d About an hour ago 549MB centos latest 5d0da3dc9764 2 months ago 231MB
8 Import and Export of Mirrors
We can push the image into the mirror warehouse on Host 1, then pull the image down on Host 2, which is a bit of a hassle. If I'm just testing it, I can run on another host after I've finished mirroring on one host. There's no need to push it onto the warehouse and pull it back locally.
At this point, we can package the image into a compressed file based on the existing image and copy it to another host to import it. This is the import and export function of the image.
In docker we export using docker save and import using docker load
Execute docker save export mirror on the host where the mirror has been generated
[root@Docker ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE zhaojie10/nginx v1.20.2 6f5b02e61ad4 7 minutes ago 549MB centos latest 5d0da3dc9764 2 months ago 231MB [root@Docker ~]# docker save -o nginximages.gz zhaojie10/nginx:v1.20.2 [root@Docker ~]# ls anaconda-ks.cfg nginximages.gz
Perform docker load import mirroring on another host that is not mirrored
[root@Docker ~]# ls anaconda-ks.cfg nginximages.gz [root@Docker ~]# docker load -i nginximages.gz Loaded image: zhaojie10/nginx:v1.20.2 [root@Docker ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE zhaojie10/nginx v1.20.2 6f5b02e61ad4 11 minutes ago 549MB