How Kubernetes manages storage resources.
First we will learn about Volume and how Kubernetes provides storage for containers in clusters through Volume; then we will practice several commonly used Volume types and understand their respective application scenarios; finally, we will discuss how Kubernetes separates cluster administrators from cluster users through Persistent Volume and Persistent Volume Claim, and practice Vo. Static and dynamic supply of lume.
Volume
In this section, we discuss Kubernetes'storage model Volume and learn how to map various persistent storage to containers.
We often say that containers and pods are short-lived.
The implication is that their life cycles may be short and they will be destroyed and created frequently. When the container is destroyed, the data stored in the file system inside the container will be cleared.
To persist the container's data, you can use Kubernetes Volume.
Volume's life cycle is independent of containers. Containers in Pod may be destroyed and rebuilt, but Volume will be retained.
In essence, Kubernetes Volume is a directory, similar to Docker Volume. When Volume is mount ed to Pod, all containers in Pod can access the Volume. Kubernetes Volume also supports a variety of backend types, including emptyDir, hostPath, GCE Persistent Disk, AWS Elastic Block Store, NFS, Ceph, etc. The complete list is available for reference. https://kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes
Volume provides an abstraction of various backend s. When using Volume to read and write data, the container does not need to care whether the data is stored in the file system of the local node or on the cloud hard disk. For it, all types of Volume are just a directory.
We'll start with the simplest emptyDir, Kubernetes Volume.
emptyDir
EmptyDir is the most basic Volume type. As its name indicates, an emptyDir Volume is an empty directory on the Host.
emptyDir Volume is persistent for containers, but not for Pods. When a Pod is deleted from a node, the Volume content is also deleted. But if only the container was destroyed and Pod was still there, Volume would not be affected.
That is to say, the life cycle of emptyDir Volume is the same as that of Pod.
All containers in Pod can share Volume, and they can specify their mount paths. Following is an example to practice emptyDir. The configuration file is as follows:
apiVersion: v1
kind: Pod
metadata:
name: producer-consumer
spec:
containers:
- image: busybox
name: producer
volumeMounts:
- mountPath: /producer_dir
name: shared-volume
args:
- /bin/sh
- -c
- echo "hello world"> /producer_dir/hello ; sleep 30000
- image: busybox
name: consumer
volumeMounts:
- mountPath: /consumer_dir
name: shared-volume
args:
- /bin/sh
- -c
- cat /consumer_dir/hello ; sleep 30000
volumes:
- name: shared-volume
emptyDir: {}Here we simulate a producer-consumer scenario. Pod has two containers, producer and consumer, which share a Volume. Producer is responsible for writing data to Volume, while consumer reads data from Volume.
The bottom volume of the file defines a Volume shared-volume of emptyDir type.
(2) The producer container will share-volume mount to the / producer_dir directory.
(3) The producer writes the data to the file hello through echo.
(4) The consumer container will share-volume mount to the / consumer_dir directory.
consumer reads data from the file hello through cat.
Execute the following command to create a Pod:
# kubectl apply -f emptyDir.yaml pod/producer-consumer created # kubectl get pod NAME READY STATUS RESTARTS AGE producer-consumer 2/2 Running 0 87s # kubectl logs producer-consumer consumer hello world
kubectl logs show that the container consumer successfully reads the data written by producer, verifying that the two containers share emptyDir Volume.
Because emptyDir is a directory in the Docker Host filesystem, its effect is equivalent to executing docker run-v/producer_dir and docker run-v/consumer_dir. Looking at the container configuration details through docker inspect, we found that both containers mount ed the same directory:
{
"Type": "bind",
"Source": "/var/lib/kubelet/pods/188767a3-cf18-4bf9-a89b-b0c4cf4124bb/volumes/kubernetes.io~empty-dir/shared-volume",
"Destination": "/consumer_dir",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
}
"Mounts": [
{
"Type": "bind",
"Source": "/var/lib/kubelet/pods/188767a3-cf18-4bf9-a89b-b0c4cf4124bb/volumes/kubernetes.io~empty-dir/shared-volume",
"Destination": "/producer_dir",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
Here/var/lib/kubelet/pods/3e6100eb-a97a-11e7-8f72-0800274451ad/volumes/kubernetes.io~empty-dir/shared-volume is the real path of emptyDir on Host.
EmptyDir is a temporary directory created on Host. Its advantage is that it can easily provide shared storage for containers in Pod without additional configuration. But it doesn't have persistence, and if Pod doesn't exist, emptyDir doesn't exist. Based on this feature, emptyDir is particularly suited for situations where containers in Pod need temporary shared storage space, such as the previous producer-consumer use case.
hostPath Volume.
The function of hostPath Volume is to mount the existing directory in the Docker Host file system to the container of Pod. Most applications do not use hostPath Volume because it actually increases the coupling between Pod and nodes, limiting the use of Pod. However, applications that need access to Kubernetes or Docker internal data (configuration files and binary libraries) need to use hostPath.
For example, kube-apiserver and kube-controller-manager are such applications, through which
kubectl edit --namespace=kube-system pod kube-apiserver-k8s-master
To see the configuration of kube-apiserver Pod, the following is the relevant section of Volume:
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /etc/pki
name: etc-pki
readOnly: true
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
volumes:
- hostPath:
path: /etc/ssl/certs
type: DirectoryOrCreate
name: ca-certs
- hostPath:
path: /etc/pki
type: DirectoryOrCreate
name: etc-pki
- hostPath:
path: /etc/kubernetes/pki
type: DirectoryOrCreate
name: k8s-certsThree hostPath volume k8s, certs and PKI are defined, which correspond to the Host directory / etc/kubernetes, / etc/ssl/certs and / etc/pki, respectively.
If the Pod is destroyed, the directory corresponding to the hostpath will also be preserved. From this point of view, the persistence of hostpath is better than emptyDir. But once Host crashes, hostPath can't be accessed.
A truly persistent Volume.
External Storage Provider
If Kubernetes is deployed on public clouds such as AWS, GCE, Azure, you can directly use the cloud hard disk as Volume. Here is an example of AWS Elastic Block Store:
To use ESB volume in Pod, you must first create it in AWS and then refer to it through volume-id. The use of other cloud hard drives can be referred to the official documents of various public cloud vendors.
Kubernetes Volume can also use mainstream distributed storage, such as Ceph, GlusterFS, etc. the following is an example of Ceph:
Ceph's / some/path/in/side/cephfs directory is mount ed to the container path / test-ceph.
Compared with emptyDir and hostPath, the biggest feature of these Volume types is that they do not rely on Kubernetes. Volume's underlying infrastructure is managed by an independent storage system, separated from the Kubernetes cluster. After data is persisted, even the entire Kubernetes crash will not be compromised.
Of course, storage systems such as operations and maintenance are usually not a simple task, especially when there are high requirements for reliability, high availability and scalability.
Volume provides a very good data persistence solution, but there are still shortcomings in manageability. In the next section, we'll learn about storage solutions with more executive rationality: Persistent Volume & Persistent Volume Claim.
PV & PVC
Volume provides a very good data persistence solution, but there are still shortcomings in manageability.
Take the previous AWS EBS example. To use Volume, Pod must know the following information beforehand:
- Currently Volume comes from AWS EBS.
- EBS Volume has been created beforehand and knows the exact volume-id.
Pod is usually maintained by the application developer, while Volume is usually maintained by the administrator of the storage system. Developers need to get the above information:
- Or ask the administrator.
- Either they are administrators themselves.
This leads to a management problem: the responsibilities of application developers and system administrators are coupled. It is acceptable if the system is small or for a development environment. However, when the cluster size becomes larger, especially for the generating environment, considering efficiency and security, this has become a problem that must be solved.
Kubernetes's solution is Persistent Volume and Persistent Volume Claim.
Persistent volume (PV) is a piece of storage space in the external storage system, which is created and maintained by the administrator. Like Volume, PV is persistent and lifecycle independent of Pod.
Persistent volume claim (PVC) is a claim for PV. PVC is usually created and maintained by ordinary users. When storage resources need to be allocated to Pod, users can create a PVC that specifies the size of storage resources and access modes (such as read-only), and Kubernetes will find and provide PVs that meet the requirements.
With Persistent Volume Claim, users only need to tell Kubernetes what kind of storage resources they need, without having to care about where the real space is allocated and how to access the underlying details. The underlying information of the Storage Provider is handled by the administrator, who should only care about the details of creating Persistent Volume.
Kubernetes supports a variety of Persistent Volumes, such as AWS EBS, Ceph, NFS, etc. For a complete list, see https://kubernetes.io/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes
NFS PersistentVolume
Practice PV and PVC through NFS.
As a preparation, we have built an NFS server on the k8s-master node with the directory / nfsdata:
[root@k8s-master ~]# showmount -e Export list for k8s-master: /nfsdata 192.168.168.0/24
Next, create a PV mypv1 configuration file nfs-pv1.yml as follows:
apiVersion: v1
kind: PersistentVolume
metadata:
name: mypv1
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: nfs
nfs:
path: /nfsdata/pv1
server: 192.168.168.10(1) Capacity specifies that the capacity of PV is 1G.
(2) accessModes specifies the access mode as ReadWriteOnce, and supports the following access modes:
ReadWriteOnce-PV can mount to a single node in read-write mode.
ReadOnlyMany-PV can mount to multiple nodes in read-only mode.
ReadWriteMany-PV can mount to multiple nodes in read-write mode.
(3) The persistent Volume Reclaim Policy specifies that when the PV recovery strategy is Recycle, the supporting strategies are:
Retain - Manual recycling by administrators is required.
Recycle --- Clear the data in PV, the effect is equivalent to executing rm-rf/the volume/*.
Delete - Delete the corresponding storage resources on Storage Provider, such as AWS EBS, GCE PD, Azure Disk, OpenStack Cinder Volume, etc.
Storage ClassName specifies that the class of PV is nfs. It is equivalent to setting up a classification for PV. PVC can specify the PV of the corresponding class to apply for.
(5) Specify the corresponding directory of PV on the NFS server.
Create mypv1:
[root@k8s-master ~]# kubectl apply -f nfs-pv1.yml persistentvolume/mypv1 created [root@k8s-master ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE mypv1 1Gi RWO Recycle Available nfs 7s
STATUS is Available, indicating that mypv1 is ready and can be applied for by PVC.
Next, create PVC mypvc1. The configuration file nfs-pvc1.yml is as follows:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mypvc1
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: nfsPVC is very simple, just specify the PV capacity, access mode and class.
Create mypvc1:
[root@k8s-master ~]# kubectl apply -f nfs-pvc1.yml persistentvolumeclaim/mypvc1 created [root@k8s-master ~]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mypvc1 Bound mypv1 1Gi RWO nfs 6s [root@k8s-master ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE mypv1 1Gi RWO Recycle Bound default/mypvc1 nfs 2m18s [root@k8s-master ~]#
From the output of kubectl get pvc and kubectl get pv, we can see that mypvc1 has Bound to mypv1, and the application is successful.
Next you can use storage in Pod. The Pod configuration file pod1.yml is as follows:
kind: Pod
apiVersion: v1
metadata:
name: mypod1
spec:
containers:
- name: mypod1
image: busybox
args:
- /bin/sh
- -c
- sleep 30000
volumeMounts:
- mountPath: "/mydata"
name: mydata
volumes:
- name: mydata
persistentVolumeClaim:
claimName: mypvc1Similar to the normal Volume format, in volumes, you specify the Volume to be applied for using mypvc1 through persistent Volume Claim.
Create mypod1:
[root@k8s-master ~]# kubectl apply -f pod1.yml pod/mypod1 created [root@k8s-master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mypod1 1/1 Running 0 3m19s 10.244.1.2 k8s-node1 <none> <none>
Verify that PV is available:
[root@k8s-master ~]# kubectl exec mypod1 touch /mydata/hello [root@k8s-master ~]# ll /nfsdata/pv1/ total 0 -rw-r--r-- 1 root root 0 Oct 12 16:36 hello
As you can see, the file / mydata/hello created in Pod has indeed been saved in the NFS server directory / nfsdata/pv1.
If you no longer need to use PV, you can remove PVC and recycle PV.
How does MySQL use PV and PVC?
This section demonstrates how to provide persistent storage for MySQL databases by:
Create PV and PVC.
Deploy MySQL.
Add data to MySQL.
Kubernetes automatically migrates MySQL to other nodes to simulate node downtime.
Verify data consistency.
First, create PV and PVC with the following configuration:
mysql-pv.yml
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-pv
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 1Gi
persistentVolumeReclaimPolicy: Retain
storageClassName: nfs
nfs:
path: /nfsdata/mysql-pv
server: 192.168.77.10mysql-pvc.yml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mysql-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: nfsCreate mysql-pv and mysql-pvc:
Next, deploy MySQL with the following configuration files:
apiVersion: v1
kind: Service
metadata:
name: mysql
spec:
ports:
- port: 3306
selector:
app: mysql
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: mysql
spec:
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- image: mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pvc PVC mysql-pvc Bound PV mysql-pv will be mount ed to MySQL data directory var/lib/mysql.
MySQL is deployed to k8s-node 2, and the following Service mysql is accessed through the client:
# kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppassword
Update the database:
[root@k8s-master ~]# kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppassword If you don't see a command prompt, try pressing enter. mysql> use mysql Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> create table my_id(id int(4)); Query OK, 0 rows affected (0.09 sec) mysql> insert into my_id values(111); Query OK, 1 row affected (0.00 sec) mysql> select * from my_id; +------+ | id | +------+ | 111 | +------+ 1 row in set (0.00 sec)
Switch to database mysql.
(2) Create the database table my_id.
(3) Insert a piece of data.
(4) Verify that the data has been written.
Turn k8s-node 2 off to simulate node downtime.
# systemctl poweroff
Over time, Kubernetes migrated MySQL to k8s-node 1.
[root@k8s-master ~]# kubectl get pod NAME READY STATUS RESTARTS AGE mysql-84bdf65dd5-bjz8b 1/1 Terminating 0 22m mysql-84bdf65dd5-ddlhc 1/1 Running 0 34s
Verify data consistency:
[root@k8s-master ~]# kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppassword If you don't see a command prompt, try pressing enter. mysql> use mysql Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> select * from my_id; +------+ | id | +------+ | 111 | +------+ 1 row in set (0.01 sec) mysql> quit
MySQL service is restored and the data is intact.
Volumes of emptyDir and hostPath types are convenient, but not durable. Kubernetes supports Volumes of various external storage systems.
PV and PVC separate the responsibilities of administrators and ordinary users, and are more suitable for production environment.