This article is just a note.

scene

Observation of CPU usage of processes

Observe CPU usage of functions in the process:

sudo perf top -p <pid>

Display function call chain at the same time:

sudo perf top -g -p <pid>

Sampling results are recorded for subsequent analysis, and - g records the call chain:

sudo perf record -g -p <pid>

Read the sampling results:

sudo perf report

Observing CPU usage of process in container

The processes in the container can actually be seen on the host machine, and ps-ef | grep < text > can be found.

So we can also use perf top-p < PID > to observe, but there will be the problem that function symbols can not be displayed. Pay attention to the bottom line of perf top:

Failed to open /opt/bitnami/php/lib/php/extensions/opcache.so, continuing without symbols

The solution is to record the sampled data with perf record, then bind the file system in the container to host, and then specify the symbol directory with perf report -- symfs < path >. You have to install bindfs first.

mkdir /tmp/foo
PID=$(docker inspect --format {{.State.Pid}} <container-name>)
bindfs /proc/$PID/root /tmp/foo
perf report --symfs /tmp/foo

# Don't forget to unbind after use
umount /tmp/foo/

Change the <container-name> above to the container name you want to observe.

Observe CPU usage of Java processes

You have to install it first. perf-map-agent (There are installation methods below), add the - XX:+PreserveFramePointer parameter when starting the Java process, and here are several uses:

• perf-java-top <pid> <perf-top-options>
• perf-java-record-stack <pid> <perf-record-options>
• perf-java-report-stack <pid> <perf-report-options>

See the official website for more information.

You can also use perf-java-flames < PID > < perf-record-options > to generate flame diagrams. You have to install them first. FlameGraph (There are installation methods below). On the Interpretation of Flame Diagram netflix blog.

Observe CPU usage of Java processes in containers

At present, there is no way.

Appendix: Installation Method

The following are the installation methods on Ubuntu 16.04 system.

perf

Install perf

$ sudo apt install -y linux-tools-common

Running perf will result in:

$ perf
WARNING: perf not found for kernel 4.4.0-145

  You may need to install the following packages for this specific kernel:
    linux-tools-4.4.0-145-generic
    linux-cloud-tools-4.4.0-145-generic

  You may also want to install one of the following packages to keep up to date:
    linux-tools-generic
    linux-cloud-tools-generic

So install:

sudo apt install linux-tools-4.4.0-145-generic linux-cloud-tools-4.4.0-145-generic linux-cloud-tools-generic

bindfs

reach bindfs official website Download the source package (version 1.13.11 in this article).

First install the tools needed for compilation:

sudo apt install -y cmake pkg-config libfuse-dev libfuse2 autoconf 

Unzip the source package, enter the bindfs directory, compile:

./configure && make && sudo make install

perf-map-agent

reach github Source repository of clone perf-map-agent.

Install JDK, which is used to start all the programs you want to monitor later. This JDK is also used to compile perf-map-agent. The way to install openjdk with apt is shown below.

Compile:

cmake .
make

# will create links to run scripts in /usr/local/bin
sudo bin/create-links-in /usr/local/bin

Install openjdk

sudo apt-get install -y openjdk-8-jdk

There is no JAVA_HOME environment variable for installation in this way, so we need to set up our own installation path to find openjdk:

dpkg-query -L openjdk-8-jdk

Write the findings in ~/.bashrc:

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

FlameGraph

reach github clone FlameGraph's source repository.

To ~/.bashrc, set the environment variables:

export FLAMEGRAPH_DIR=<path-to-flame-graph>