Using cloud disks to store volumes often requires a cloud disk of appropriate capacity when the service is initialized, but as data grows, the capacity of the data disk cannot meet demand and needs to be expanded.

Kubernetes is an automated scheduling and scheduling system that implements life cycle management of data volumes.In K8S 1.14, CSI data volume expansion is in the Alpha phase and requires Feature Gates to be turned on for use.

This paper describes how to dynamically expand cloud disks in a CSI environment:

Instructions:

1. Data backup:

Keep in mind: Make a snapshot backup of the cloud disk before you expand the volume to prevent data problems caused by abnormal expansion process;

2. Cluster dependency:

3. Data volume limits:

3. Requirements for StorageClass:

Dependent preparation

Apply for ACK cluster (greater than or equal to version 1.14) Aliyun Kubernetes cluster (select CSI storage plug-in when applying for cluster);

1. Configure Feature Gate (for K8S1.14 cluster):

Since resize is also the Feature of Alpha in K8S 1.14, the following configuration needs to be added:

Update kube-controller-manager Add Feature Gate:

/etc/kubernetes/manifests/kube-controller-manager.yaml

Update kubelet (scriptable if there are many nodes):

/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
systemctl daemon-reload
service kubelet restart

feature gates: --feature-gates=ExpandCSIVolumes=true

2. Cluster Add Extension Rights:

Expanding cloud disks requires adding ResizeDisk permissions to the Worker RAM role of the cluster:

Private Clusters:
In Cluster --> Management --> Cluster Resources Click on the Master RAM Role; Edit Ram permissions, add ResizeDisk as follows:

Hosted Cluster:
In Cluster --> Management --> Cluster Resource Click on the Worker RAM Role; Edit Ram permissions, add ResizeDisk as follows:

3. resizer plug-in deployment (for K8S1.14 cluster):

Refer to the following template:

kind: Service
apiVersion: v1
metadata:
  name: csi-resizer
  namespace: kube-system
  labels:
    app: csi-resizer
spec:
  selector:
    app: csi-resizer
  ports:
    - name: dummy
      port: 12345
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: csi-resizer
  namespace: kube-system
spec:
  serviceName: "csi-resizer"
  selector:
    matchLabels:
      app: csi-resizer
  template:
    metadata:
      labels:
        app: csi-resizer
    spec:
      tolerations:
      - operator: "Exists"
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: node-role.kubernetes.io/master
                operator: Exists
      priorityClassName: system-node-critical
      serviceAccount: admin
      hostNetwork: true
      containers:
        - name: csi-resizer
          image: registry.cn-hangzhou.aliyuncs.com/acs/csi-resizer:v0.3.0
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
            - "--leader-election"
          env:
            - name: ADDRESS
              value: /socketDir/csi.sock
          imagePullPolicy: "Always"
          volumeMounts:
            - name: socket-dir
              mountPath: /socketDir/

        - name: csi-diskplugin
          securityContext:
            privileged: true
            capabilities:
              add: ["SYS_ADMIN"]
            allowPrivilegeEscalation: true
          image: registry.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.14.8.32-c77e277b-aliyun
          imagePullPolicy: "Always"
          args:
            - "--endpoint=$(CSI_ENDPOINT)"
            - "--v=5"
            - "--driver=diskplugin.csi.alibabacloud.com"
          env:
            - name: CSI_ENDPOINT
              value: unix://socketDir/csi.sock
          volumeMounts:
            - mountPath: /var/log/
              name: host-log
            - mountPath: /socketDir/
              name: socket-dir
            - name: etc
              mountPath: /host/etc

      volumes:
        - name: socket-dir
          emptyDir: {}
        - name: host-log
          hostPath:
            path: /var/log/
        - name: etc
          hostPath:
            path: /etc
  updateStrategy:
    type: RollingUpdate

Cloud roll expansion:

1. Create an application

Create a nginx application and mount a 20G cloud disk data volume to Pod with the following templates for PVC and Deploy:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-disk
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: alicloud-disk-ssd

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dynamic-create
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        volumeMounts:
          - name: disk-pvc
            mountPath: "/data"
      volumes:
        - name: disk-pvc
          persistentVolumeClaim:
            claimName: pvc-disk

The current application status is as follows:

Pod The mounted cloud disk size is 20 G；
# kubectl get pod
NAME                              READY   STATUS    RESTARTS   AGE
dynamic-create-857bd875b5-n82d4   1/1     Running   0          107s
# kubectl exec -ti dynamic-create-857bd875b5-n82d4 df | grep data
/dev/vdb        20511312   45080  20449848   1% /data


pvc,pv All sizes are shown as 20 G；
# kubectl get pvc
NAME       STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS        AGE
pvc-disk   Bound    d-wz9g8sl8dl1ks8hz2m82   20Gi       RWO            alicloud-disk-ssd   2m17s

# kubectl get pv
NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM              STORAGECLASS        REASON   AGE
d-wz9g8sl8dl1ks8hz2m82   20Gi       RWO            Delete           Bound    default/pvc-disk   alicloud-disk-ssd            2m15s

2. Cloud roll expansion:

Expand Cloud Disk to perform the following commands:

# kubectl patch pvc pvc-disk -p '{"spec":{"resources":{"requests":{"storage":"30Gi"}}}}'

Updating the pvc size will drive Resizer to call the cloud disk api for expansion. The console can check that the cloud disk has become 30G and the size of the pv has been updated to 30G.

# kubectl get pvc
NAME       STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS        AGE
pvc-disk   Bound    d-wz9g8sl8dl1ks8hz2m82   20Gi       RWO            alicloud-disk-ssd   13m

# kubectl get pv
NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM              STORAGECLASS        REASON   AGE
d-wz9g8sl8dl1ks8hz2m82   30Gi       RWO            Delete           Bound    default/pvc-disk   alicloud-disk-ssd            13m

At this time, only cloud disk expansion has been completed, file system expansion has not been done, so the storage space in the container is still 20G;

# kubectl exec -ti dynamic-create-857bd875b5-n82d4 df /data
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/vdb        20511312 45080  20449848   1% /data

Trigger file system expansion by deleting Pod:

# kubectl delete pod dynamic-create-857bd875b5-n82d4
pod "dynamic-create-857bd875b5-n82d4" deleted

# kubectl get pod
NAME                              READY   STATUS    RESTARTS   AGE
dynamic-create-857bd875b5-4gng9   1/1     Running   0          38s

//Visible file system has been expanded to 30G:
# kubectl exec -ti dynamic-create-857bd875b5-4gng9 df /data
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/vdb        30832548 45036  30771128   1% /data

The above steps complete the steps for cloud disk expansion in a CSI environment: