Learn these tips and let the Go program monitor itself

When it comes to allowing the Go program to monitor the resource usage of its own process, let's talk about what indicators need to be monitored. Generally speaking, the most common indicators of the process are the memory occupation rate, CPU occupation rate and the number of threads created. Because the go language maintains goroutines on the thread itself, the resource index for the go process also needs to add the number of goroutines created.

Because many services are deployed on Kubernetes clusters, a Go process is often a Pod, but the resources of the container are shared with the host machine, and the upper limit of its resources is specified when it is created. Therefore, the specific situation needs to be discussed separately when obtaining CPU and Memory information.

How to use Go to obtain various indicators of the process

Let's first discuss how to obtain these indicators in the case of ordinary host and virtual machine. The container environment will be discussed in the next section.

The gopstuil library can be used to obtain the resource usage of the Go process. It shields the differences between various systems and helps us easily obtain various system and hardware information. gopsutil divides different functions into different sub packages. The modules it provides mainly include:

CPU: system CPU related modules;
Disk: system disk related modules;
Docker: docker related modules;
mem: memory related modules;
net: network related;
Process: process related modules;
winservices: Windows service related modules.

We only use its process sub package to obtain process related information.

Statement: the process module needs to be imported into the project after "GitHub. COM / shirou / gopsutil / process". The os and other modules used in the later demonstration code will uniformly omit the import related information and error handling, which will be explained in advance here.

Create process object

The NewProcess of the process module will return a process object holding the specified PID. The method will check whether the PID exists. If it does not exist, an error will be returned. We can obtain various information about the process through other methods defined on the process object.

p, _ := process.NewProcess(int32(os.Getpid()))

CPU usage of the process

The CPU utilization rate of a process needs to be calculated by calculating the CPU utilization time change of the process within the specified time

cpuPercent, err := p.Percent(time.Second)

The above returns the proportion of all CPU time. If you want to see the proportion more intuitively, you can calculate the proportion of a single core.

cp := cpuPercent / float64(runtime.NumCPU())

Memory usage, number of threads, and number of goroutine s

The acquisition of these three indicators is too simple. Let's put them together

// Gets the proportion of memory occupied by the process
mp, _ := p.MemoryPercent()
// Number of threads created
threadCount := pprof.Lookup("threadcreate").Count()
// Goroutine number 
gNum := runtime.NumGoroutine()

The above method to obtain the proportion of process resources can be accurate only in the virtual machine and physical machine environment. Linux containers like Docker rely on Linux Namespace and Cgroups technology to realize process isolation and resource limitation, which is not feasible.

Many companies now deploy services in K8s clusters. Therefore, if the resource usage of Go processes is obtained in Docker, it needs to be calculated according to the upper limit of resources allocated to containers by Cgroups.

Get process indicators in container environment

In Linux, the operating interface exposed by Cgroups to users is the file system. It is organized in the form of files and directories in the / sys/fs/cgroup path of the operating system. There are many subdirectories such as cpuset, cpu and memory under / sys/fs/cgroup. Each subdirectory represents the resource types that can be limited by Cgroups.

For our need to monitor Go process memory and CPU indicators, we only need to know cpu.cfs_period_us,cpu.cfs_quota_us and memory.limit_ in_ Just bytes. The first two parameters need to be used in combination and can be used to limit the length of the process to cfs_period can only be allocated to CFS in total_ The CPU time of quota can be simply understood as the number of cores that can be used by the container = cfs_quota / cfs_period.

Therefore, the method of obtaining the CPU proportion of Go process in the container needs to be adjusted. Use the formula given above to calculate the maximum number of cores that the container can use.

cpuPeriod, err := readUint("/sys/fs/cgroup/cpu/cpu.cfs_period_us")

cpuQuota, err := readUint("/sys/fs/cgroup/cpu/cpu.cfs_quota_us")

cpuNum := float64(cpuQuota) / float64(cpuPeriod)

Then divide the proportion of the process obtained through p.Percent occupying all the CPU time of the machine by the calculated number of cores to calculate the proportion of the Go process to the CPU in the container.

cpuPercent, err := p.Percent(time.Second)
// cp := cpuPercent / float64(runtime.NumCPU())
// Adjust to
cp := cpuPercent / cpuNum

The maximum memory that can be used by the container is naturally in memory.limit_in_bytes, so the proportion of memory occupied by the Go process in the container needs to be obtained through the following method

memLimit, err := readUint("/sys/fs/cgroup/memory/memory.limit_in_bytes")
memInfo, err := p.MemoryInfo
mp := memInfo.RSS * 100 / memLimit

RSS in the above process memory information is called resident memory, which is the amount of memory allocated to the process in RAM and allowed to be accessed by the process. The readUint used to read container resources is the method given by the containerd organization in the cgroups implementation.

func readUint(path string) (uint64, error) {
 v, err := ioutil.ReadFile(path)
 if err != nil {
  return 0, err
 }
 return parseUint(strings.TrimSpace(string(v)), 10, 64)
}

func parseUint(s string, base, bitSize int) (uint64, error) {
 v, err := strconv.ParseUint(s, base, bitSize)
 if err != nil {
  intValue, intErr := strconv.ParseInt(s, base, bitSize)
  // 1. Handle negative values greater than MinInt64 (and)
  // 2. Handle negative values lesser than MinInt64
  if intErr == nil && intValue < 0 {
   return 0, nil
  } else if intErr != nil &&
   intErr.(*strconv.NumError).Err == strconv.ErrRange &&
   intValue < 0 {
   return 0, nil
  }
  return 0, err
 }
 return v, nil
}

I will give a link to their source code in the reference link below.

Reference link

Contianerd utils: https://github.com/containerd/cgroups/blob/318312a373405e5e91134d8063d04d59768a1bff/utils.go#L243
What is RSS: https://stackoverflow.com/questions/7880784/what-is-rss-and-vsz-in-linux-memory-management

- END -

Posted by GremlinP1R on Wed, 17 Nov 2021 22:06:42 -0800

Programmer Group