Small pit record of flink performance monitoring based on Prometheus

Keywords: Java Big Data flink


The flink cluster in the company has been running for a long time and has not been fully monitored. Recently, it plans to do this. Through online research, the deployment mode adopted by the company is per job mode. Finally, the job index is pushed to the push gateway of the intermediate gateway based on prometheus, Then prometheus grabs the information on the push gateway to monitor the performance of the flick. Finally, it is displayed through Grafana.


During the access process, I found a problem: I feel that the mechanism of prometheus is a little stupid!!!
Why do you say that?
Pushgateway will create a group for the indicator data pushed by the taskmanager and jobmanager of each flyjob. prometheus will pull the latest data of a batch of groups from the pushgateway every other cycle. It seems that there is no problem, but when a flick job is rescheduled or restarted, a new job group will be generated in the push gateway, and the old group will never be deleted.
prometheus, no matter whether your group is expired or not, it has no brain to regularly pull data from all registered groups, which means that those expired groups will always pull the last duplicate data.

Of course, I am also thinking that even if the pushgateway does not manage the cleaning of expired groups, should the flyk actively delete groups at least during the end of the job?
After entering the source code of Prometheus pushgateway reporter, I found that, ah, there are really:

    public void close() {
        if (deleteOnShutdown && pushGateway != null) {
            try {
                pushGateway.delete(jobName, groupingKey);
            } catch (IOException e) {
                        "Failed to delete metrics from PushGateway with jobName {}, groupingKey {}.",

However, in fact, this method will not be called at the end of the job. I don't know if it's a flick bug (my flick version is 1.13)?

cheat your papa!!!


It is conceivable that if the oil is drained like this, it will expand to what extent..

Therefore, there is the following script to clean up automatically and regularly:

trap 'echo "got sigterm" ; exit 0' SIGTERM


#function convert_to_standardnotation(){
#    # convert number from scientific notation to standar d( ie  '1.5383780136826127e+09' )
#    printf '%.0f' $1

function convert_to_standardnotation() {
    # convert number from scientific notation to standard( ie  '1.5383780136826127e+09' )
    echo $1 | awk '{printf("%.0f", $1)}'

function extract_pushgateway_variable(){
 local -r _METRIC=$1
 local -r _VARNAME=$2
 #echo 'push_time_seconds{instance="",job="bk_jenkins"} 1.5383802210997093e+09' | sed -r 's/.*instance="([^"]*).*/\1/g'
 echo $_METRIC | sed -r "s/.*${_VARNAME}=\"([^\"]*).*/\\1/g"
 # sample usage :
 # extract_pushgateway_variable 'push_time_seconds{instance="",job="bk_jenkins"} 1.5383802210997093e+09' 'instance'

function check_metric_line(){
   local -r _line=$1
   METRIC_TIME=$(echo $_line | awk '{print $2}' )
   #echo "mtime = $_line -> $METRIC_TIME "
   METRIC_TIME=$(convert_to_standardnotation $METRIC_TIME)


    metricInstance=$(extract_pushgateway_variable "$_line" 'instance')
    metricJob=$(extract_pushgateway_variable "$_line" 'job')
    echo "[INFO] job should be deleted $metricJob  - $metricInstance  age: $METRIC_AGE_SECONDS "
  #  curl -s -X DELETE "$PGW_URL/metrics/job/${metricJob}/instance/${metricInstance}"
    curl -s -X DELETE "$PGW_URL/metrics/job/${metricJob}"


function check_expired_metric_loop(){

export CURRENT_TIME=$(date +%s)
METRICS_LIST=$(curl -s  $PGW_URL/metrics | egrep "^push_time_seconds")
echo "$METRICS_LIST" | while  read -r line || [[ -n "$line" ]]; do
   check_metric_line "$line"
sleep $((EXPIRATION_SECONDS / 3 ))

while : ; do

Only replace PGW when using_ The URL address is enough

In addition, this alternative method can also be tried, but I think it's too troublesome, so I'll post it here to record:

Posted by jtgraphic on Fri, 26 Nov 2021 16:43:59 -0800