background

The flink cluster in the company has been running for a long time and has not been fully monitored. Recently, it plans to do this. Through online research, the deployment mode adopted by the company is per job mode. Finally, the job index is pushed to the push gateway of the intermediate gateway based on prometheus, Then prometheus grabs the information on the push gateway to monitor the performance of the flick. Finally, it is displayed through Grafana.

problem

During the access process, I found a problem: I feel that the mechanism of prometheus is a little stupid!!!
Why do you say that?
Pushgateway will create a group for the indicator data pushed by the taskmanager and jobmanager of each flyjob. prometheus will pull the latest data of a batch of groups from the pushgateway every other cycle. It seems that there is no problem, but when a flick job is rescheduled or restarted, a new job group will be generated in the push gateway, and the old group will never be deleted.
prometheus, no matter whether your group is expired or not, it has no brain to regularly pull data from all registered groups, which means that those expired groups will always pull the last duplicate data.

Of course, I am also thinking that even if the pushgateway does not manage the cleaning of expired groups, should the flyk actively delete groups at least during the end of the job?
After entering the source code of Prometheus pushgateway reporter, I found that, ah, there are really:

   @Override
    public void close() {
        if (deleteOnShutdown && pushGateway != null) {
            try {
                pushGateway.delete(jobName, groupingKey);
            } catch (IOException e) {
                log.warn(
                        "Failed to delete metrics from PushGateway with jobName {}, groupingKey {}.",
                        jobName,
                        groupingKey,
                        e);
            }
        }
        super.close();
    }

However, in fact, this method will not be called at the end of the job. I don't know if it's a flick bug (my flick version is 1.13)?

cheat your papa!!!

solve

It is conceivable that if the oil is drained like this, it will expand to what extent..

Therefore, there is the following script to clean up automatically and regularly:

trap 'echo "got sigterm" ; exit 0' SIGTERM

EXPIRATION_SECONDS=${EXPIRATION_SECONDS:-900}
PGW_URL=${PGW_URL:-http://192.168.1.1:9091}

#function convert_to_standardnotation(){
#    # convert number from scientific notation to standar d( ie  '1.5383780136826127e+09' )
#    printf '%.0f' $1
#}

function convert_to_standardnotation() {
    # convert number from scientific notation to standard( ie  '1.5383780136826127e+09' )
    echo $1 | awk '{printf("%.0f", $1)}'
}

function extract_pushgateway_variable(){
 local -r _METRIC=$1
 local -r _VARNAME=$2
 #echo 'push_time_seconds{instance="10.32.32.7",job="bk_jenkins"} 1.5383802210997093e+09' | sed -r 's/.*instance="([^"]*).*/\1/g'
 echo $_METRIC | sed -r "s/.*${_VARNAME}=\"([^\"]*).*/\\1/g"
 # sample usage :
 # extract_pushgateway_variable 'push_time_seconds{instance="10.32.32.7",job="bk_jenkins"} 1.5383802210997093e+09' 'instance'
}

function check_metric_line(){
   local -r _line=$1
   METRIC_TIME=$(echo $_line | awk '{print $2}' )
   #echo "mtime = $_line -> $METRIC_TIME "
   METRIC_TIME=$(convert_to_standardnotation $METRIC_TIME)
   #echo "$CURRENT_TIME - $METRIC_TIME "
   METRIC_AGE_SECONDS=$((CURRENT_TIME-METRIC_TIME))

   if [ "$METRIC_AGE_SECONDS" -gt "$EXPIRATION_SECONDS" ]; then

    metricInstance=$(extract_pushgateway_variable "$_line" 'instance')
    metricJob=$(extract_pushgateway_variable "$_line" 'job')
    
    echo "[INFO] job should be deleted $metricJob  - $metricInstance  age: $METRIC_AGE_SECONDS "
  #  curl -s -X DELETE "$PGW_URL/metrics/job/${metricJob}/instance/${metricInstance}"
    curl -s -X DELETE "$PGW_URL/metrics/job/${metricJob}"
     fi


}


function check_expired_metric_loop(){

export CURRENT_TIME=$(date +%s)
METRICS_LIST=$(curl -s  $PGW_URL/metrics | egrep "^push_time_seconds")
echo "$METRICS_LIST" | while  read -r line || [[ -n "$line" ]]; do
   check_metric_line "$line"
done
sleep $((EXPIRATION_SECONDS / 3 ))

}
while : ; do
check_expired_metric_loop
done 

Only replace PGW when using_ The URL address is enough

In addition, this alternative method can also be tried, but I think it's too troublesome, so I'll post it here to record:

https://blog.csdn.net/daijiguo/article/details/105453643