Originally, I wanted to use CEPH fuse, but as a result, I always reported mount can't read super Block error, but I only installed the rpm package of CEPH fuse for the host. kublet is actually using the kernel mount mode to mount cephfs. This is a mistake that can't be repeated. But how to specify kubelet to use CEPH fuse to mount cephfs? I google d for a long time and nobody talked about this small problem. Later, I had no choice but to turn to the source code directly: kubernetes/pkg/volume/cephfs/cephfs.go

	// check whether it belongs to fuse, if not, default to use kernel mount.
	if cephfsVolume.checkFuseMount() {
		klog.V(4).Info("CephFS fuse mount.")
		err = cephfsVolume.execFuseMount(dir)
		// cleanup no matter if fuse mount fail.
		keyringPath := cephfsVolume.GetKeyringPath()
		_, StatErr := os.Stat(keyringPath)
		if !os.IsNotExist(StatErr) {
			os.RemoveAll(keyringPath)
		}
		if err == nil {
			// cephfs fuse mount succeeded.
			return nil
		}
		// if cephfs fuse mount failed, fallback to kernel mount.
		klog.V(2).Infof("CephFS fuse mount failed: %v, fallback to kernel mount.", err)

	}
	klog.V(4).Info("CephFS kernel mount.")

	err = cephfsVolume.execMount(dir)

This is very clear. By default, check CEPH fuse first, and then directly use CEPH fuse to mount. If the mount fails or the check fails, try again by using kernel mount. That is to say, the reason why kubelet uses kernel mount may be that CEPH fuse check fails or mount fails. Let's have a look at how it checks

func (cephfsVolume *cephfsMounter) checkFuseMount() bool {
	execute := cephfsVolume.plugin.host.GetExec(cephfsVolume.plugin.GetPluginName())
	switch runtime.GOOS {
	case "linux":
		if _, err := execute.Command("/usr/bin/test", "-x", "/sbin/mount.fuse.ceph").CombinedOutput(); err == nil {
			klog.V(4).Info("/sbin/mount.fuse.ceph exists, it should be fuse mount.")
			return true
		}
		return false
	}
	return false
}

Just judge whether / sbin/mount.fuse.ceph can be executed. Look for the log

kubelet: I0424 17:09:38.366917    7829 reconciler.go:252] operationExecutor.MountVolume started for volume "xxx" (UniqueName: "kubernetes.io/cephfs/xxxx") pod "xxxx" (UID: "xxx")
kubelet: I0424 17:09:38.632672    7829 cephfs.go:259] CephFS fuse mount failed: Ceph-fuse failed: signal: aborted
kubelet: arguments: [-k /data/kubelet/pods/9db514c0-0d0b-4343-ad0f-ff71abe9d38d/volumes/kubernetes.io~cephfs/xxx
-pro~keyring/admin.keyring -m xxxx
olumes/kubernetes.io~cephfs/xxxx -r /production --id admin]
kubelet: Output: 2020-04-24 17:09:38.375590 7fa188e78ec0 -1 did not load config file, using default settings.
kubelet: 2020-04-24 17:09:38.393668 7fa188e78ec0 -1 init, newargv = 0x557071540720 newargc=11
kubelet: ceph-fuse[1256508]: starting ceph client
kubelet: ceph-fuse[1256508]: starting fuse
kubelet: , fallback to kernel mount.
systemd: Started Kubernetes transient mount for /data/kubelet/pods/xxx/volumes/kubernetes.io~cephfs/xx.
kernel: libceph: mon1 xxx session established
kernel: libceph: xx fsid xxx
kernel: ceph: problem parsing mds trace -5
kernel: ceph: mds parse_reply err -5
kernel: ceph: mdsc_handle_reply got corrupt reply mds0(tid:1)
kubelet: E0424 17:09:38.967287    7829 mount_linux.go:140] Mount failed: exit status 32

From the log point of view, there is something wrong with the mount, but there is no problem with my manual mount. The only difference is the parameters, - k-m. I judge that the old version of CEPH fuse does not support these parameters. I just need to upgrade CEPH fuse to the latest version, which is also the case. I didn't expect there were still these pits here