Problem: as soon as the kfana container built by k8s is restarted, the configured dashboard and added users will be reset. As a result, there is no data persistence in grafana.
Solve:
1. Establish persistent flower storage pvc in the cluster first
Then hang it in the container / var directory
volumeMounts:
- mountPath: /var
name: grafana-storage
volumes:
- name: grafana-storage
persistentVolumeClaim:
claimName: grafana
A lot of information on the Internet is done in this way. However, it doesn't work. Restart still loses configuration.

2. Referring to many materials, it is found that it is necessary to support mysql data test to realize grafana data persistence.
For MySQL installation, you can use helm to install through helm search mysql. MySQL needs to hang persistent storage, which is not covered here.
The following details how to configure mysql database by grafana.
First of all, create a new grafana database in mysql, and refer to the mysql sql syntax
Secondly:
Step 1: find out the node where the old grafana container is located through k8s cluster, find the container, and export grafana.ini to the server.
docker cp ad7fb5f9ef38:/etc/grafana/grafana.ini /lyf/
Step 2: edit the / lyf/grafana.ini file and modify the database option

	[database]
	\# You can configure the database connection by specifying type, host, name, user and password
	\# as seperate properties or as on string using the url propertie.

	\# Either "mysql", "postgres" or "sqlite3", it's your choice
	type = mysql
	host = mysql.default:3306
	name = grafana
	user = root
	\# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
	password = root.123

After modifying the configuration file, save it. Then execute the following command to create a configmap
Create configmap grafana etc
kubectl create configmap "grafana-etc" --from-file=/lyf/grafana.ini --namespace=kube-system

Step 3: create grafana deployment
[root@k8s01 prometheus]# cat grafana-deploy.yaml

	apiVersion: extensions/v1beta1
	kind: Deployment
	metadata:
	  annotations:
	    deployment.kubernetes.io/revision: "15"
	  labels:
	    k8s-app: grafana
	    task: monitoring
	  name: monitoring-grafana
	  namespace: kube-system
	spec:
	  progressDeadlineSeconds: 600
	  replicas: 1
	  revisionHistoryLimit: 10
	  selector:
	    matchLabels:
	      k8s-app: grafana
	      task: monitoring
	  strategy:
	    rollingUpdate:
	      maxSurge: 1
	      maxUnavailable: 1
	    type: RollingUpdate
	  template:
	    metadata:
	      labels:
	        k8s-app: grafana
	        task: monitoring
	    spec:
	      containers:
	      - env:
	        - name: INFLUXDB_HOST
	          value: monitoring-influxdb.kube-system
	        - name: GF_SERVER_HTTP_PORT
	          value: "3000"
	        - name: GF_AUTH_BASIC_ENABLED
	          value: "false"
	        - name: GF_AUTH_ANONYMOUS_ENABLED
	          value: "false"
	        - name: GF_AUTH_ANONYMOUS_ORG_ROLE
	          value: Admin
	        - name: GF_SERVER_ROOT_URL
	          value: /
	        image: registry.gcloud.srcb.com/registry.cn-hangzhou.aliyuncs.com/acs/grafana:5.0.4-gpu-monitoring
	        imagePullPolicy: IfNotPresent
	        name: grafana
	        ports:
	        - containerPort: 3000
	          protocol: TCP
	        resources: {}
	        terminationMessagePath: /dev/termination-log
	        terminationMessagePolicy: File
	        volumeMounts:
	        - mountPath: /etc/ssl/certs
	          name: ca-certificates
	          readOnly: true
	        - mountPath: /var
	          name: grafana-storage
	        - mountPath: /etc/grafana/
	          name: grafana-etc-volume
	      dnsPolicy: ClusterFirst
	      nodeSelector:
	        role: master
	      restartPolicy: Always
	      schedulerName: default-scheduler
	      securityContext: {}
	      terminationGracePeriodSeconds: 30
	      volumes:
	      - hostPath:
	          path: /etc/ssl/certs
	          type: ""
	        name: ca-certificates
	      - name: grafana-storage
	        persistentVolumeClaim:
	          claimName: grafana
	      - configMap:
	          defaultMode: 420
	          items:
	          - key: grafana.ini
	            path: grafana.ini
	          name: grafana-etc
	        name: grafana-etc-volume

Summary: restart configuration of grafana container with mysql configuration will not be lost.
Reference resources: https://blog.csdn.net/wenwst/article/details/76624019