• At first, the privilege mode of apiserver, kubelet node is opened in accordance with the relevant documents - allow-privileged=true, and then executed with ali document, it can not run at all. Referring to issue in github, we know that it's all about why the problem doesn't work, but the response is not clear, and it does mention the need to support fuse on the host.

• Start self-debugging: unable to search for relevant information, download source code, compile and check
  git clone https://github.com/denverdino/lxcfs-initializer.git

• Dockfile also knows that the libraries and other files can not meet the needs of its own version. At present, we just need to let him run, and then execute the script content of start.sh inside, to see where the specific error is?

• The content of lxcfs-image/Dockerfile according to your environment change is as follows

FROM daocloud.io/centos:7.3.1611
RUN yum -y install fuse fuse-devel pam-devel wget install gcc automake autoconf libtool make
ENV LXCFS_VERSION 2.0.8
RUN wget https://linuxcontainers.org/downloads/lxcfs/lxcfs-$LXCFS_VERSION.tar.gz && \
mkdir /lxcfs && tar xzvf lxcfs-$LXCFS_VERSION.tar.gz -C /lxcfs --strip-components=1 && \
cd /lxcfs && ./configure && make && make install
STOPSIGNAL SIGINT
ADD start.sh /
CMD ["/bin/sleep","10000"]

• Build lxcfs: sleep image

  [root@ns-yun-020037 ~]# cd lxcfs-initializer/
  docker build -t lxcfs:sleep lxcfs-image

• Change the mirror name to lxcfs:sleep according to the yaml file of the original daemonSet

• Enter the node container location problem, according to the / start.sh script to execute the relevant commands, you can see that the last step of execution can not find the file lxcfs

[root@yun-020040 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4e1cb10dd73e xxx:80/test/lxcfs "/bin/sleep 10000" 52 seconds ago Up 51 seconds k8s_lxcfs_lxcfs-4m5g7_default_b1306fd2-3bd4-11e9-bb5d-ec388f7928b2_0

[root@yun-020040 ~]# docker exec -it 4e1cb10dd73e /bin/bash 
[root@lxcfs-4m5g7 /]#
[root@lxcfs-4m5g7 /]# nsenter -m/proc/1/ns/mnt fusermount -u /var/lib/lxcfs 2> /dev/null || true
[root@lxcfs-4m5g7 /]# nsenter -m/proc/1/ns/mnt [ -L /etc/mtab ] ||sed -i "/^lxcfs \/var\/lib\/lxcfs fuse.lxcfs/d" /etc/mtab
[root@lxcfs-4m5g7 /]# mkdir -p /usr/local/lib/lxcfs /var/lib/lxcfs
[root@lxcfs-4m5g7 /]# exec nsenter -m/proc/1/ns/mnt lxcfs /var/lib/lxcfs/
nsenter: failed to execute lxcfs: No such file or directory

• According to the content of Dockerfile, the container should have it.
  https://github.com/denverdino/lxcfs-initializer/blob/master/lxcfs-image/Dockerfile

• Start directly with docker to see if there is a problem. Execute the start.sh command, and it can be executed normally.

  [root@yun-020040 ~]# docker  run --privileged=true -it lxcfs:sleep  /bin/bash
  [root@10ca4ad41ce4 /]# nsenter -m/proc/1/ns/mnt fusermount -u /var/lib/lxcfs 2> /dev/null || true
  [root@10ca4ad41ce4 /]# nsenter -m/proc/1/ns/mnt [ -L /etc/mtab ] ||sed -i "/^lxcfs \/var\/lib\/lxcfs fuse.lxcfs/d" /etc/mtab
  [root@10ca4ad41ce4 /]# mkdir -p /usr/local/lib/lxcfs /var/lib/lxcfs
  [root@10ca4ad41ce4 /]# exec nsenter -m/proc/1/ns/mnt lxcfs /var/lib/lxcfs/
  hierarchies:
  0: fd: 5: perf_event
  1: fd: 6: hugetlb
  2: fd: 7: pids
  3: fd: 8: cpuacct,cpu
  4: fd: 9: blkio
  5: fd: 10: devices
  6: fd: 11: cpuset
  7: fd: 12: memory
  8: fd: 13: freezer
  9: fd: 14: net_prio,net_cls
  10: fd: 15: name=systemd

• Looking back at the yaml file of k8s, there are files that are mounted in the host's / usr/local directory and are hosts, as shown in bold below.

volumeMounts:
- name: cgroup
mountPath: /sys/fs/cgroup
- name: lxcfs
mountPath: /var/lib/lxcfs
mountPropagation: Bidirectional
- name: usr-local
**mountPath: /usr/local**
volumes:
- name: cgroup
hostPath:
path: /sys/fs/cgroup
**- name: usr-local**
hostPath:
path: /usr/local
- name: lxcfs
hostPath:
path: /var/lib/lxcfs
type: DirectoryOrCreate

• Remove the / usr/local mount from the yaml file and see if the lxcfs file inside the container can be used properly. It has been proved that it failed.

• It is suggested that lxcfs be re-installed and deployed on the host computer, and then daemonSet is run. The program runs normally.

Test results:
This project only hosts the host startup process to daemonSet for unified management. The host also has to provide lib lib libraries and other related binary files.

• Further analysis of his init container content, main.go, facilitates the mounting of directories.

  flag.StringVar(&annotation, "annotation", defaultAnnotation, "The annotation to trigger initialization")
  flag.StringVar(&initializerName, "initializer-name", defaultInitializerName, "The initializer name")
  flag.StringVar(&namespace, "namespace", "default", "The configuration namespace")
  flag.BoolVar(&requireAnnotation, "require-annotation", true, "Require annotation for initialization")
  flag.Parse()
  
  log.Println("Starting the Kubernetes initializer...")
  log.Printf("Initializer name set to: %s", initializerName)
  
  clusterConfig, err := rest.InClusterConfig()
  if err != nil {
      log.Fatal(err.Error())
  }
  
  clientset, err := kubernetes.NewForConfig(clusterConfig)
  if err != nil {
      log.Fatal(err)
  }
  
  // -v /var/lib/lxcfs/proc/cpuinfo:/proc/cpuinfo:rw
  // -v /var/lib/lxcfs/proc/diskstats:/proc/diskstats:rw
  // -v /var/lib/lxcfs/proc/meminfo:/proc/meminfo:rw
  // -v /var/lib/lxcfs/proc/stat:/proc/stat:rw
  // -v /var/lib/lxcfs/proc/swaps:/proc/swaps:rw
  // -v /var/lib/lxcfs/proc/uptime:/proc/uptime:rw
  c := &config{
      volumeMounts: []corev1.VolumeMount{
          corev1.VolumeMount{
              Name:      "lxcfs-proc-cpuinfo",
              MountPath: "/proc/cpuinfo",
          },

Reference material:
https://www.alibabacloud.com/blog/kubernetes-demystified%3A-using-lxcfs-to-improve-container-resource-visibility_594109?spm=a2c41.12195345.0.0
https://github.com/denverdino/lxcfs-initializer