Interpretation of core classes of XXL job basic components II: JobTriggerPoolHelper

Keywords: Java xxl-job

preface:

        In the process of specifically interpreting the functional principles of the core classes of JobTriggerPoolHelper, please familiarize yourself with the basic principles of ThreadPoolExecutor, volatile, ConcurrentMap, AtomicInteger, etc. at the same time, please consider the problems of JobTriggerPoolHelper class during program operation when cluster deployment is adopted? Tips: volatile, AtomicInteger, ConcurrentMap.

Summary:

        This time, the JobTriggerPoolHelper class will be parsed through the execution of a task. The execution process of a simple periodic task will be observed through the JobScheduleHelper class and XxlJobTrigger class. The volatile keyword and the basic classes ConcurrentMap and AtomicInteger are concatenated.

Volatile keyword reference address

volatile principle:

For a variable modified by volatile keyword, the compiler and runtime will notice that the variable is shared, so the operations on the variable will not be reordered together with other memory operations. Volatile variables are not cached in registers or invisible to other processors, so the latest written value is always returned when reading variables of volatile type.

Main functions of JobScheduleHelper class: task delivery and next execution time maintenance

      step1: scan every 5 seconds. If the condition is less than (current time + 5 seconds) the task to be, the query quantity is calculated according to the minimum default value, and the result is 6000 (minimum value) each time

      Step 2: it is judged that the execution is missed, and the task is delivered to the thread pool by calling JobTriggerPoolHelper.trigger method

      step3: update the next execution time of the task

Main functions of JobTriggerPoolHelper class: thread pool asynchronously triggers tasks

  Interpretation 1: thread pool isolation -- speed separation

Purpose: some slow executing threads will slow down the whole thread pool, so we need to separate the speed from the slow.
It is necessary to distinguish which are slow threads. The basis here is that the number of slow executions (time-consuming more than 500ms) in a minute is 10.

Note:  

      1. The mintim variable and jobTimeoutCountMap variable are decorated with volatile keyword, but in the cluster environment, the count here will be inaccurate. Of course, it has no impact on the execution of tasks, but the time of entering the slow thread pool may be delayed, which reduces the probability

    2. Under what circumstances will the slow thread pool be accessed:

            1. The task execution frequency is high, and the execution times per minute should be more than 10 times (jobTimeoutCountMap, which stores the slow execution times of each task, and automatically empties the container after 60 seconds)

            2. The job client takes a long time to execute tasks, causing accumulation or serious network delay

            3. Database related queries are time-consuming and encounter bottlenecks such as 100% CPU or no connection

  3. Add the trigger task to the thread pool: the core method is the same as the execution logic of the fast and slow thread pool (the difference is the capacity of the queue)  

JobTriggerPoolHelper class reference source code:

/**
 * job trigger thread pool helper
 *
 * @author xuxueli 2018-07-03 21:08:07
 */
public class JobTriggerPoolHelper {
    private static Logger logger = LoggerFactory.getLogger(JobTriggerPoolHelper.class);


    // ---------------------- trigger pool ----------------------

    // fast/slow thread pool
    private ThreadPoolExecutor fastTriggerPool = null;
    private ThreadPoolExecutor slowTriggerPool = null;

    public void start(){
        fastTriggerPool = new ThreadPoolExecutor(
                10,
                XxlJobAdminConfig.getAdminConfig().getTriggerPoolFastMax(),
                60L,
                TimeUnit.SECONDS,
                new LinkedBlockingQueue<Runnable>(1000),
                new ThreadFactory() {
                    @Override
                    public Thread newThread(Runnable r) {
                        return new Thread(r, "xxl-job, admin JobTriggerPoolHelper-fastTriggerPool-" + r.hashCode());
                    }
                });

        slowTriggerPool = new ThreadPoolExecutor(
                10,
                XxlJobAdminConfig.getAdminConfig().getTriggerPoolSlowMax(),
                60L,
                TimeUnit.SECONDS,
                new LinkedBlockingQueue<Runnable>(2000),
                new ThreadFactory() {
                    @Override
                    public Thread newThread(Runnable r) {
                        return new Thread(r, "xxl-job, admin JobTriggerPoolHelper-slowTriggerPool-" + r.hashCode());
                    }
                });
    }


    public void stop() {
        //triggerPool.shutdown();
        fastTriggerPool.shutdownNow();
        slowTriggerPool.shutdownNow();
        logger.info(">>>>>>>>> xxl-job trigger thread pool shutdown success.");
    }


    // job timeout count
    private volatile long minTim = System.currentTimeMillis()/60000;     // ms > min
    private volatile ConcurrentMap<Integer, AtomicInteger> jobTimeoutCountMap = new ConcurrentHashMap<>();


    /**
     * add trigger
     */
    public void addTrigger(final int jobId,
                           final TriggerTypeEnum triggerType,
                           final int failRetryCount,
                           final String executorShardingParam,
                           final String executorParam,
                           final String addressList,boolean isUnactive) {

        // choose thread pool
        ThreadPoolExecutor triggerPool_ = fastTriggerPool;
        AtomicInteger jobTimeoutCount = jobTimeoutCountMap.get(jobId);
        if (jobTimeoutCount!=null && jobTimeoutCount.get() > 10) {      // job-timeout 10 times in 1 min
            triggerPool_ = slowTriggerPool;
        }

        // trigger
        triggerPool_.execute(new Runnable() {
            @Override
            public void run() {

                long start = System.currentTimeMillis();

                try {
                    // do trigger
                    // Start scheduling
                    XxlJobTrigger.trigger(jobId, triggerType, failRetryCount, executorShardingParam, executorParam, addressList,isUnactive);
                } catch (Exception e) {
                    logger.error(e.getMessage(), e);
                } finally {

                    // check timeout-count-map
                    long minTim_now = System.currentTimeMillis()/60000;
                    if (minTim != minTim_now) {
                        minTim = minTim_now;
                        jobTimeoutCountMap.clear();
                    }

                    // incr timeout-count-map
                    long cost = System.currentTimeMillis()-start;
                    if (cost > 500) {       // ob-timeout threshold 500ms
                        AtomicInteger timeoutCount = jobTimeoutCountMap.putIfAbsent(jobId, new AtomicInteger(1));
                        if (timeoutCount != null) {
                            timeoutCount.incrementAndGet();
                        }
                    }

                }

            }
        });
    }



    // ---------------------- helper ----------------------

    private static JobTriggerPoolHelper helper = new JobTriggerPoolHelper();

    public static void toStart() {
        helper.start();
    }
    public static void toStop() {
        helper.stop();
    }

    /**
     * @param jobId
     * @param triggerType
     * @param failRetryCount
     * 			>=0: use this param
     * 			<0: use param from job info config
     * @param executorShardingParam
     * @param executorParam
     *          null: use job param
     *          not null: cover job param
     */
    public static void trigger(int jobId, TriggerTypeEnum triggerType, int failRetryCount, String executorShardingParam, String executorParam, String addressList,boolean isUnactive) {
        helper.addTrigger(jobId, triggerType, failRetryCount, executorShardingParam, executorParam, addressList,isUnactive);
    }

}

    XxlJobTrigger class function: execute task trigger method

         Interpretation of trigger method:

                 step1. Initialize the relevant parameters during task execution and the execution method: routing policy and fragment execution

                 Step 2. Call the function processTrigger that is actually processed and scheduled to trigger the task  

 

            Interpretation of processTrigger method:

                   Step 1: compete for distributed locks. Only after obtaining the lock can the service continue to execute

                   Step 2: create execution log + select the address of the executor

                  step3: trigger task execution through http client (if the task method is synchronous, the time-consuming point will be formed, the number of times will be accumulated, and the condition for the task to be delivered to the slow thread pool will be formed)

    Refer to the source code of XxlJobTrigger class: (the lock competition part is changed to redis and zookeeper)

/**
 * xxl-job trigger
 * Created by xuxueli on 17/7/13.
 */
public class XxlJobTrigger {
    private static Logger logger = LoggerFactory.getLogger(XxlJobTrigger.class);
    /**
     * Default lock type
     */
    private static final String DEFAULT_LOCK_TYPE = "redis";

    /**
     * trigger job
     *
     * @param jobId
     * @param triggerType
     * @param failRetryCount        >=0: use this param
     *                              <0: use param from job info config
     * @param executorShardingParam
     * @param executorParam         null: use job param
     *                              not null: cover job param
     * @param addressList           null: use executor addressList
     *                              not null: cover
     */
    public static void trigger(int jobId,
                               TriggerTypeEnum triggerType,
                               int failRetryCount,
                               String executorShardingParam,
                               String executorParam,
                               String addressList, boolean isUnactive) {

        // load data
        XxlJobInfo jobInfo = XxlJobAdminConfig.getAdminConfig().getXxlJobInfoDao().loadById(jobId);
        if (isUnactive) {
            jobInfo.setTriggerLastTime(System.currentTimeMillis());
        }
        if (jobInfo == null) {
            logger.warn(">>>>>>>>>>>> trigger fail, jobId invalid,jobId={}", jobId);
            return;
        }
        int finalFailRetryCount = failRetryCount >= 0 ? failRetryCount : jobInfo.getExecutorFailRetryCount();
        XxlJobGroup group = XxlJobAdminConfig.getAdminConfig().getXxlJobGroupDao().load(jobInfo.getJobGroup());

        // cover addressList
        if (addressList != null && addressList.trim().length() > 0) {
            group.setAddressType(1);
            group.setAddressList(addressList.trim());
        }

        // sharding param
        int[] shardingParam = null;
        if (executorShardingParam != null) {
            String[] shardingArr = executorShardingParam.split("/");
            if (shardingArr.length == 2 && isNumeric(shardingArr[0]) && isNumeric(shardingArr[1])) {
                shardingParam = new int[2];
                shardingParam[0] = Integer.valueOf(shardingArr[0]);
                shardingParam[1] = Integer.valueOf(shardingArr[1]);
            }
        }
        if (ExecutorRouteStrategyEnum.SHARDING_BROADCAST == ExecutorRouteStrategyEnum.match(jobInfo.getExecutorRouteStrategy(), null)
                && group.getRegistryList() != null && !group.getRegistryList().isEmpty()
                && shardingParam == null) {
            for (int i = 0; i < group.getRegistryList().size(); i++) {
                processTrigger(group, jobInfo, finalFailRetryCount, triggerType, i, group.getRegistryList().size());
            }
        } else {
            if (shardingParam == null) {
                shardingParam = new int[]{0, 1};
            }
            processTrigger(group, jobInfo, finalFailRetryCount, triggerType, shardingParam[0], shardingParam[1]);
        }

    }

    private static boolean isNumeric(String str) {
        try {
            int result = Integer.valueOf(str);
            return true;
        } catch (NumberFormatException e) {
            return false;
        }
    }

    /**
     * TODO Functions that actually handle and schedule
     *
     * @param group               job group, registry list may be empty
     * @param jobInfo
     * @param finalFailRetryCount
     * @param triggerType
     * @param index               sharding index
     * @param total               sharding index
     */
    private static void processTrigger(XxlJobGroup group, XxlJobInfo jobInfo, int finalFailRetryCount, TriggerTypeEnum triggerType, int index, int total) {
        boolean result = false;

        // Judge whether it belongs to API calling mode
        if (!TriggerTypeEnum.API.equals(jobInfo.getRealTriggerType())) {
            String lockType = XxlJobAdminConfig.getAdminConfig().getLockType();
            long time = System.currentTimeMillis() / 1000;

            String currentTag = jobInfo.getId() + "-" + jobInfo.getExecutorParam();
            if (!DEFAULT_LOCK_TYPE.equals(lockType)) {
                result = ZookeeperUtils.lock("/job-id-" + currentTag, jobInfo.getJobDesc());
                logger.info("xxl job seize zk Distributed lock ");
            } else {
                String uid = UUID.randomUUID().toString();
                /// RedisLockUtil.tryLock(jobInfo.getId()+"-" + time,uid,5000,0,0);
                result = RedisUtils.lock(currentTag, uid, 100);
                logger.info("xxl job seize redis Distributed lock ");
            }

            if (!result) {
                logger.info("xxl job Failed to preempt distributed lock! lock=" + currentTag + "-" + time);
                return;
            }
            logger.info("xxl job Successfully preempted distributed lock! lock=" + currentTag + "-" + time);
        }

        // param
        // block strategy
        ExecutorBlockStrategyEnum blockStrategy = ExecutorBlockStrategyEnum.match(jobInfo.getExecutorBlockStrategy(), ExecutorBlockStrategyEnum.SERIAL_EXECUTION);
        // route strategy
        ExecutorRouteStrategyEnum executorRouteStrategyEnum = ExecutorRouteStrategyEnum.match(jobInfo.getExecutorRouteStrategy(), null);
        String shardingParam = (ExecutorRouteStrategyEnum.SHARDING_BROADCAST == executorRouteStrategyEnum) ? String.valueOf(index).concat("/").concat(String.valueOf(total)) : null;

        // 1,save log-id
        XxlJobLog jobLog = new XxlJobLog();
        jobLog.setJobGroup(jobInfo.getJobGroup());
        jobLog.setJobId(jobInfo.getId());
        jobLog.setTriggerTime(new Date());
        XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().save(jobLog);
        logger.debug(">>>>>>>>>>> xxl-job trigger start, jobId:{}", jobLog.getId());

        // 2,init trigger-param
        ExpandTriggerParam triggerParam = new ExpandTriggerParam();
        triggerParam.setJobId(jobInfo.getId());
        triggerParam.setExecutorHandler(jobInfo.getExecutorHandler());
        triggerParam.setExecutorParams(jobInfo.getExecutorParam());
        triggerParam.setExecutorBlockStrategy(jobInfo.getExecutorBlockStrategy());
        triggerParam.setExecutorTimeout(jobInfo.getExecutorTimeout());
        triggerParam.setLogId(jobLog.getId());
        triggerParam.setLogDateTime(jobLog.getTriggerTime().getTime());
        triggerParam.setGlueType(jobInfo.getGlueType());
        triggerParam.setGlueSource(jobInfo.getGlueSource());
        triggerParam.setGlueUpdatetime(jobInfo.getGlueUpdatetime().getTime());
        triggerParam.setBroadcastIndex(index);
        triggerParam.setBroadcastTotal(total);
        // Latest execution time (refer to those requiring accurate time)
        triggerParam.setTriggerLastTime(jobInfo.getTriggerLastTime());

        // 3,init address
        String address = null;
        ReturnT<String> routeAddressResult = null;
        if (group.getRegistryList() != null && !group.getRegistryList().isEmpty()) {
            if (ExecutorRouteStrategyEnum.SHARDING_BROADCAST == executorRouteStrategyEnum) {
                if (index < group.getRegistryList().size()) {
                    address = group.getRegistryList().get(index);
                } else {
                    address = group.getRegistryList().get(0);
                }
            } else {
                routeAddressResult = executorRouteStrategyEnum.getRouter().route(triggerParam, group.getRegistryList());
                if (routeAddressResult.getCode() == ReturnT.SUCCESS_CODE) {
                    address = routeAddressResult.getContent();
                }
            }
        } else {
            routeAddressResult = new ReturnT<String>(ReturnT.FAIL_CODE, I18nUtil.getString("jobconf_trigger_address_empty"));
        }

        // 4,trigger remote executor
        ReturnT<String> triggerResult = null;
        if (address != null) {
            triggerResult = runExecutor(triggerParam, address);
        } else {
            triggerResult = new ReturnT<String>(ReturnT.FAIL_CODE, null);
        }

        // 5,collection trigger info
        StringBuilder triggerMsgSb = new StringBuilder();
        triggerMsgSb.append(I18nUtil.getString("jobconf_trigger_type")).append(": ").append(triggerType.getTitle());
        triggerMsgSb.append("<br>").append(I18nUtil.getString("jobconf_trigger_admin_adress")).append(": ").append(IpUtil.getIp());
        triggerMsgSb.append("<br>").append(I18nUtil.getString("jobconf_trigger_exe_regtype")).append(": ")
                .append((group.getAddressType() == 0) ? I18nUtil.getString("jobgroup_field_addressType_0") : I18nUtil.getString("jobgroup_field_addressType_1"));
        triggerMsgSb.append("<br>").append(I18nUtil.getString("jobconf_trigger_exe_regaddress")).append(": ").append(group.getRegistryList());
        triggerMsgSb.append("<br>").append(I18nUtil.getString("jobinfo_field_executorRouteStrategy")).append(": ").append(executorRouteStrategyEnum.getTitle());
        if (shardingParam != null) {
            triggerMsgSb.append("(").append(shardingParam).append(")");
        }
        triggerMsgSb.append("<br>").append(I18nUtil.getString("jobinfo_field_executorBlockStrategy")).append(": ").append(blockStrategy.getTitle());
        triggerMsgSb.append("<br>").append(I18nUtil.getString("jobinfo_field_timeout")).append(": ").append(jobInfo.getExecutorTimeout());
        triggerMsgSb.append("<br>").append(I18nUtil.getString("jobinfo_field_executorFailRetryCount")).append(": ").append(finalFailRetryCount);

        triggerMsgSb.append("<br><br><span style=\"color:#00c0ef;\" > >>>>>>>>>>>").append(I18nUtil.getString("jobconf_trigger_run")).append("<<<<<<<<<<< </span><br>")
                .append((routeAddressResult != null && routeAddressResult.getMsg() != null) ? routeAddressResult.getMsg() + "<br><br>" : "").append(triggerResult.getMsg() != null ? triggerResult.getMsg() : "");

        // 6,save log trigger-info
        jobLog.setExecutorAddress(address);
        jobLog.setExecutorHandler(jobInfo.getExecutorHandler());
        jobLog.setExecutorParam(jobInfo.getExecutorParam());
        jobLog.setExecutorShardingParam(shardingParam);
        jobLog.setExecutorFailRetryCount(finalFailRetryCount);

        // The special rule retry logic triggers the retry caused by the failure of the client request
        if (triggerResult.getCode() != ReturnT.SUCCESS_CODE && !jobInfo.getExecutorFailRetryStrategy().equals(ExecutorRetryStrategyEnum.DEFAULT_RETRY.name()) && jobLog.getExecutorFailRetryCount() > 0) {
            MqMessageSender mqMessageSender = XxlJobAdminConfig.getAdminConfig().getMqMessageSender();
            String regexTime = jobInfo.getExecutorFailRetryInterval();
            if (jobInfo.getExecutorFailRetryStrategy().equals(ExecutorRetryStrategyEnum.REGEX_RETRY.name())) {
                int idx = jobInfo.getExecutorFailRetryCount() - jobLog.getExecutorFailRetryCount();
                String[] regexs = jobInfo.getExecutorFailRetryRegex().split(",");
                // Eliminate empty string
                List<String> regexList = Arrays.stream(regexs).filter(StrUtil::isNotBlank).collect(Collectors.toList());
                if (index < regexList.size()) {
                    regexTime = regexList.get(idx);
                }
            }
            if (!StringUtil.isNullOrEmpty(regexTime)) {
                mqMessageSender.sendDeliverTime(XxlJobAdminConfig.getAdminConfig().getXxlJobMqConfig().getRetryTopic(), (jobLog.getId() + "").getBytes(), regexTime, null);
                jobLog.setExecutorFailRetryCount(jobLog.getExecutorFailRetryCount() - 1);
            }
        }
        /// jobLog.setTriggerTime();
        jobLog.setTriggerCode(triggerResult.getCode());
        jobLog.setTriggerMsg(triggerMsgSb.toString());
        XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().updateTriggerInfo(jobLog);
        /// RedisLockUtil.releaseLock(jobInfo.getId()+"",uid);
        logger.debug(">>>>>>>>>>> xxl-job trigger end, jobId:{}", jobLog.getId());
    }

    /**
     * run executor
     *
     * @param triggerParam
     * @param address
     * @return
     */
    public static ReturnT<String> runExecutor(ExpandTriggerParam triggerParam, String address) {
        ReturnT<String> runResult = null;
        try {
            ExecutorBiz executorBiz = XxlJobScheduler.getExecutorBiz(address);
            runResult = executorBiz.run(triggerParam);
        } catch (Exception e) {
            logger.error(">>>>>>>>>>> xxl-job trigger error, please check if the executor[{}] is running.", address, e);
            runResult = new ReturnT<String>(ReturnT.FAIL_CODE, ThrowableUtil.toString(e));
        }

        StringBuffer runResultSB = new StringBuffer(I18nUtil.getString("jobconf_trigger_run") + ": ");
        runResultSB.append("<br>address: ").append(address);
        runResultSB.append("<br>code: ").append(runResult.getCode());
        runResultSB.append("<br>msg: ").append(runResult.getMsg());

        runResult.setMsg(runResultSB.toString());
        return runResult;
    }

}

Posted by Sentosa on Sat, 06 Nov 2021 19:51:37 -0700