preface:
In the process of specifically interpreting the functional principles of the core classes of JobTriggerPoolHelper, please familiarize yourself with the basic principles of ThreadPoolExecutor, volatile, ConcurrentMap, AtomicInteger, etc. at the same time, please consider the problems of JobTriggerPoolHelper class during program operation when cluster deployment is adopted? Tips: volatile, AtomicInteger, ConcurrentMap.
Summary:
This time, the JobTriggerPoolHelper class will be parsed through the execution of a task. The execution process of a simple periodic task will be observed through the JobScheduleHelper class and XxlJobTrigger class. The volatile keyword and the basic classes ConcurrentMap and AtomicInteger are concatenated.
Volatile keyword reference address
volatile principle:
For a variable modified by volatile keyword, the compiler and runtime will notice that the variable is shared, so the operations on the variable will not be reordered together with other memory operations. Volatile variables are not cached in registers or invisible to other processors, so the latest written value is always returned when reading variables of volatile type.
Main functions of JobScheduleHelper class: task delivery and next execution time maintenance
step1: scan every 5 seconds. If the condition is less than (current time + 5 seconds) the task to be, the query quantity is calculated according to the minimum default value, and the result is 6000 (minimum value) each time
Step 2: it is judged that the execution is missed, and the task is delivered to the thread pool by calling JobTriggerPoolHelper.trigger method
step3: update the next execution time of the task
Main functions of JobTriggerPoolHelper class: thread pool asynchronously triggers tasks
Interpretation 1: thread pool isolation -- speed separation
Purpose: some slow executing threads will slow down the whole thread pool, so we need to separate the speed from the slow.
It is necessary to distinguish which are slow threads. The basis here is that the number of slow executions (time-consuming more than 500ms) in a minute is 10.
Note:
1. The mintim variable and jobTimeoutCountMap variable are decorated with volatile keyword, but in the cluster environment, the count here will be inaccurate. Of course, it has no impact on the execution of tasks, but the time of entering the slow thread pool may be delayed, which reduces the probability
2. Under what circumstances will the slow thread pool be accessed:
1. The task execution frequency is high, and the execution times per minute should be more than 10 times (jobTimeoutCountMap, which stores the slow execution times of each task, and automatically empties the container after 60 seconds)
2. The job client takes a long time to execute tasks, causing accumulation or serious network delay
3. Database related queries are time-consuming and encounter bottlenecks such as 100% CPU or no connection
3. Add the trigger task to the thread pool: the core method is the same as the execution logic of the fast and slow thread pool (the difference is the capacity of the queue)
JobTriggerPoolHelper class reference source code:
/**
* job trigger thread pool helper
*
* @author xuxueli 2018-07-03 21:08:07
*/
public class JobTriggerPoolHelper {
private static Logger logger = LoggerFactory.getLogger(JobTriggerPoolHelper.class);
// ---------------------- trigger pool ----------------------
// fast/slow thread pool
private ThreadPoolExecutor fastTriggerPool = null;
private ThreadPoolExecutor slowTriggerPool = null;
public void start(){
fastTriggerPool = new ThreadPoolExecutor(
10,
XxlJobAdminConfig.getAdminConfig().getTriggerPoolFastMax(),
60L,
TimeUnit.SECONDS,
new LinkedBlockingQueue<Runnable>(1000),
new ThreadFactory() {
@Override
public Thread newThread(Runnable r) {
return new Thread(r, "xxl-job, admin JobTriggerPoolHelper-fastTriggerPool-" + r.hashCode());
}
});
slowTriggerPool = new ThreadPoolExecutor(
10,
XxlJobAdminConfig.getAdminConfig().getTriggerPoolSlowMax(),
60L,
TimeUnit.SECONDS,
new LinkedBlockingQueue<Runnable>(2000),
new ThreadFactory() {
@Override
public Thread newThread(Runnable r) {
return new Thread(r, "xxl-job, admin JobTriggerPoolHelper-slowTriggerPool-" + r.hashCode());
}
});
}
public void stop() {
//triggerPool.shutdown();
fastTriggerPool.shutdownNow();
slowTriggerPool.shutdownNow();
logger.info(">>>>>>>>> xxl-job trigger thread pool shutdown success.");
}
// job timeout count
private volatile long minTim = System.currentTimeMillis()/60000; // ms > min
private volatile ConcurrentMap<Integer, AtomicInteger> jobTimeoutCountMap = new ConcurrentHashMap<>();
/**
* add trigger
*/
public void addTrigger(final int jobId,
final TriggerTypeEnum triggerType,
final int failRetryCount,
final String executorShardingParam,
final String executorParam,
final String addressList,boolean isUnactive) {
// choose thread pool
ThreadPoolExecutor triggerPool_ = fastTriggerPool;
AtomicInteger jobTimeoutCount = jobTimeoutCountMap.get(jobId);
if (jobTimeoutCount!=null && jobTimeoutCount.get() > 10) { // job-timeout 10 times in 1 min
triggerPool_ = slowTriggerPool;
}
// trigger
triggerPool_.execute(new Runnable() {
@Override
public void run() {
long start = System.currentTimeMillis();
try {
// do trigger
// Start scheduling
XxlJobTrigger.trigger(jobId, triggerType, failRetryCount, executorShardingParam, executorParam, addressList,isUnactive);
} catch (Exception e) {
logger.error(e.getMessage(), e);
} finally {
// check timeout-count-map
long minTim_now = System.currentTimeMillis()/60000;
if (minTim != minTim_now) {
minTim = minTim_now;
jobTimeoutCountMap.clear();
}
// incr timeout-count-map
long cost = System.currentTimeMillis()-start;
if (cost > 500) { // ob-timeout threshold 500ms
AtomicInteger timeoutCount = jobTimeoutCountMap.putIfAbsent(jobId, new AtomicInteger(1));
if (timeoutCount != null) {
timeoutCount.incrementAndGet();
}
}
}
}
});
}
// ---------------------- helper ----------------------
private static JobTriggerPoolHelper helper = new JobTriggerPoolHelper();
public static void toStart() {
helper.start();
}
public static void toStop() {
helper.stop();
}
/**
* @param jobId
* @param triggerType
* @param failRetryCount
* >=0: use this param
* <0: use param from job info config
* @param executorShardingParam
* @param executorParam
* null: use job param
* not null: cover job param
*/
public static void trigger(int jobId, TriggerTypeEnum triggerType, int failRetryCount, String executorShardingParam, String executorParam, String addressList,boolean isUnactive) {
helper.addTrigger(jobId, triggerType, failRetryCount, executorShardingParam, executorParam, addressList,isUnactive);
}
}
XxlJobTrigger class function: execute task trigger method
Interpretation of trigger method:
step1. Initialize the relevant parameters during task execution and the execution method: routing policy and fragment execution
Step 2. Call the function processTrigger that is actually processed and scheduled to trigger the task
Interpretation of processTrigger method:
Step 1: compete for distributed locks. Only after obtaining the lock can the service continue to execute
Step 2: create execution log + select the address of the executor
step3: trigger task execution through http client (if the task method is synchronous, the time-consuming point will be formed, the number of times will be accumulated, and the condition for the task to be delivered to the slow thread pool will be formed)
Refer to the source code of XxlJobTrigger class: (the lock competition part is changed to redis and zookeeper)
/**
* xxl-job trigger
* Created by xuxueli on 17/7/13.
*/
public class XxlJobTrigger {
private static Logger logger = LoggerFactory.getLogger(XxlJobTrigger.class);
/**
* Default lock type
*/
private static final String DEFAULT_LOCK_TYPE = "redis";
/**
* trigger job
*
* @param jobId
* @param triggerType
* @param failRetryCount >=0: use this param
* <0: use param from job info config
* @param executorShardingParam
* @param executorParam null: use job param
* not null: cover job param
* @param addressList null: use executor addressList
* not null: cover
*/
public static void trigger(int jobId,
TriggerTypeEnum triggerType,
int failRetryCount,
String executorShardingParam,
String executorParam,
String addressList, boolean isUnactive) {
// load data
XxlJobInfo jobInfo = XxlJobAdminConfig.getAdminConfig().getXxlJobInfoDao().loadById(jobId);
if (isUnactive) {
jobInfo.setTriggerLastTime(System.currentTimeMillis());
}
if (jobInfo == null) {
logger.warn(">>>>>>>>>>>> trigger fail, jobId invalid，jobId={}", jobId);
return;
}
int finalFailRetryCount = failRetryCount >= 0 ? failRetryCount : jobInfo.getExecutorFailRetryCount();
XxlJobGroup group = XxlJobAdminConfig.getAdminConfig().getXxlJobGroupDao().load(jobInfo.getJobGroup());
// cover addressList
if (addressList != null && addressList.trim().length() > 0) {
group.setAddressType(1);
group.setAddressList(addressList.trim());
}
// sharding param
int[] shardingParam = null;
if (executorShardingParam != null) {
String[] shardingArr = executorShardingParam.split("/");
if (shardingArr.length == 2 && isNumeric(shardingArr[0]) && isNumeric(shardingArr[1])) {
shardingParam = new int[2];
shardingParam[0] = Integer.valueOf(shardingArr[0]);
shardingParam[1] = Integer.valueOf(shardingArr[1]);
}
}
if (ExecutorRouteStrategyEnum.SHARDING_BROADCAST == ExecutorRouteStrategyEnum.match(jobInfo.getExecutorRouteStrategy(), null)
&& group.getRegistryList() != null && !group.getRegistryList().isEmpty()
&& shardingParam == null) {
for (int i = 0; i < group.getRegistryList().size(); i++) {
processTrigger(group, jobInfo, finalFailRetryCount, triggerType, i, group.getRegistryList().size());
}
} else {
if (shardingParam == null) {
shardingParam = new int[]{0, 1};
}
processTrigger(group, jobInfo, finalFailRetryCount, triggerType, shardingParam[0], shardingParam[1]);
}
}
private static boolean isNumeric(String str) {
try {
int result = Integer.valueOf(str);
return true;
} catch (NumberFormatException e) {
return false;
}
}
/**
* TODO Functions that actually handle and schedule
*
* @param group job group, registry list may be empty
* @param jobInfo
* @param finalFailRetryCount
* @param triggerType
* @param index sharding index
* @param total sharding index
*/
private static void processTrigger(XxlJobGroup group, XxlJobInfo jobInfo, int finalFailRetryCount, TriggerTypeEnum triggerType, int index, int total) {
boolean result = false;
// Judge whether it belongs to API calling mode
if (!TriggerTypeEnum.API.equals(jobInfo.getRealTriggerType())) {
String lockType = XxlJobAdminConfig.getAdminConfig().getLockType();
long time = System.currentTimeMillis() / 1000;
String currentTag = jobInfo.getId() + "-" + jobInfo.getExecutorParam();
if (!DEFAULT_LOCK_TYPE.equals(lockType)) {
result = ZookeeperUtils.lock("/job-id-" + currentTag, jobInfo.getJobDesc());
logger.info("xxl job seize zk Distributed lock ");
} else {
String uid = UUID.randomUUID().toString();
/// RedisLockUtil.tryLock(jobInfo.getId()+"-" + time,uid,5000,0,0);
result = RedisUtils.lock(currentTag, uid, 100);
logger.info("xxl job seize redis Distributed lock ");
}
if (!result) {
logger.info("xxl job Failed to preempt distributed lock! lock=" + currentTag + "-" + time);
return;
}
logger.info("xxl job Successfully preempted distributed lock! lock=" + currentTag + "-" + time);
}
// param
// block strategy
ExecutorBlockStrategyEnum blockStrategy = ExecutorBlockStrategyEnum.match(jobInfo.getExecutorBlockStrategy(), ExecutorBlockStrategyEnum.SERIAL_EXECUTION);
// route strategy
ExecutorRouteStrategyEnum executorRouteStrategyEnum = ExecutorRouteStrategyEnum.match(jobInfo.getExecutorRouteStrategy(), null);
String shardingParam = (ExecutorRouteStrategyEnum.SHARDING_BROADCAST == executorRouteStrategyEnum) ? String.valueOf(index).concat("/").concat(String.valueOf(total)) : null;
// 1,save log-id
XxlJobLog jobLog = new XxlJobLog();
jobLog.setJobGroup(jobInfo.getJobGroup());
jobLog.setJobId(jobInfo.getId());
jobLog.setTriggerTime(new Date());
XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().save(jobLog);
logger.debug(">>>>>>>>>>> xxl-job trigger start, jobId:{}", jobLog.getId());
// 2,init trigger-param
ExpandTriggerParam triggerParam = new ExpandTriggerParam();
triggerParam.setJobId(jobInfo.getId());
triggerParam.setExecutorHandler(jobInfo.getExecutorHandler());
triggerParam.setExecutorParams(jobInfo.getExecutorParam());
triggerParam.setExecutorBlockStrategy(jobInfo.getExecutorBlockStrategy());
triggerParam.setExecutorTimeout(jobInfo.getExecutorTimeout());
triggerParam.setLogId(jobLog.getId());
triggerParam.setLogDateTime(jobLog.getTriggerTime().getTime());
triggerParam.setGlueType(jobInfo.getGlueType());
triggerParam.setGlueSource(jobInfo.getGlueSource());
triggerParam.setGlueUpdatetime(jobInfo.getGlueUpdatetime().getTime());
triggerParam.setBroadcastIndex(index);
triggerParam.setBroadcastTotal(total);
// Latest execution time (refer to those requiring accurate time)
triggerParam.setTriggerLastTime(jobInfo.getTriggerLastTime());
// 3,init address
String address = null;
ReturnT<String> routeAddressResult = null;
if (group.getRegistryList() != null && !group.getRegistryList().isEmpty()) {
if (ExecutorRouteStrategyEnum.SHARDING_BROADCAST == executorRouteStrategyEnum) {
if (index < group.getRegistryList().size()) {
address = group.getRegistryList().get(index);
} else {
address = group.getRegistryList().get(0);
}
} else {
routeAddressResult = executorRouteStrategyEnum.getRouter().route(triggerParam, group.getRegistryList());
if (routeAddressResult.getCode() == ReturnT.SUCCESS_CODE) {
address = routeAddressResult.getContent();
}
}
} else {
routeAddressResult = new ReturnT<String>(ReturnT.FAIL_CODE, I18nUtil.getString("jobconf_trigger_address_empty"));
}
// 4,trigger remote executor
ReturnT<String> triggerResult = null;
if (address != null) {
triggerResult = runExecutor(triggerParam, address);
} else {
triggerResult = new ReturnT<String>(ReturnT.FAIL_CODE, null);
}
// 5,collection trigger info
StringBuilder triggerMsgSb = new StringBuilder();
triggerMsgSb.append(I18nUtil.getString("jobconf_trigger_type")).append(": ").append(triggerType.getTitle());
triggerMsgSb.append("<br>").append(I18nUtil.getString("jobconf_trigger_admin_adress")).append(": ").append(IpUtil.getIp());
triggerMsgSb.append("<br>").append(I18nUtil.getString("jobconf_trigger_exe_regtype")).append(": ")
.append((group.getAddressType() == 0) ? I18nUtil.getString("jobgroup_field_addressType_0") : I18nUtil.getString("jobgroup_field_addressType_1"));
triggerMsgSb.append("<br>").append(I18nUtil.getString("jobconf_trigger_exe_regaddress")).append(": ").append(group.getRegistryList());
triggerMsgSb.append("<br>").append(I18nUtil.getString("jobinfo_field_executorRouteStrategy")).append(": ").append(executorRouteStrategyEnum.getTitle());
if (shardingParam != null) {
triggerMsgSb.append("(").append(shardingParam).append(")");
}
triggerMsgSb.append("<br>").append(I18nUtil.getString("jobinfo_field_executorBlockStrategy")).append(": ").append(blockStrategy.getTitle());
triggerMsgSb.append("<br>").append(I18nUtil.getString("jobinfo_field_timeout")).append(": ").append(jobInfo.getExecutorTimeout());
triggerMsgSb.append("<br>").append(I18nUtil.getString("jobinfo_field_executorFailRetryCount")).append(": ").append(finalFailRetryCount);
triggerMsgSb.append("<br><br><span style=\"color:#00c0ef;\" > >>>>>>>>>>>").append(I18nUtil.getString("jobconf_trigger_run")).append("<<<<<<<<<<< </span><br>")
.append((routeAddressResult != null && routeAddressResult.getMsg() != null) ? routeAddressResult.getMsg() + "<br><br>" : "").append(triggerResult.getMsg() != null ? triggerResult.getMsg() : "");
// 6,save log trigger-info
jobLog.setExecutorAddress(address);
jobLog.setExecutorHandler(jobInfo.getExecutorHandler());
jobLog.setExecutorParam(jobInfo.getExecutorParam());
jobLog.setExecutorShardingParam(shardingParam);
jobLog.setExecutorFailRetryCount(finalFailRetryCount);
// The special rule retry logic triggers the retry caused by the failure of the client request
if (triggerResult.getCode() != ReturnT.SUCCESS_CODE && !jobInfo.getExecutorFailRetryStrategy().equals(ExecutorRetryStrategyEnum.DEFAULT_RETRY.name()) && jobLog.getExecutorFailRetryCount() > 0) {
MqMessageSender mqMessageSender = XxlJobAdminConfig.getAdminConfig().getMqMessageSender();
String regexTime = jobInfo.getExecutorFailRetryInterval();
if (jobInfo.getExecutorFailRetryStrategy().equals(ExecutorRetryStrategyEnum.REGEX_RETRY.name())) {
int idx = jobInfo.getExecutorFailRetryCount() - jobLog.getExecutorFailRetryCount();
String[] regexs = jobInfo.getExecutorFailRetryRegex().split(",");
// Eliminate empty string
List<String> regexList = Arrays.stream(regexs).filter(StrUtil::isNotBlank).collect(Collectors.toList());
if (index < regexList.size()) {
regexTime = regexList.get(idx);
}
}
if (!StringUtil.isNullOrEmpty(regexTime)) {
mqMessageSender.sendDeliverTime(XxlJobAdminConfig.getAdminConfig().getXxlJobMqConfig().getRetryTopic(), (jobLog.getId() + "").getBytes(), regexTime, null);
jobLog.setExecutorFailRetryCount(jobLog.getExecutorFailRetryCount() - 1);
}
}
/// jobLog.setTriggerTime();
jobLog.setTriggerCode(triggerResult.getCode());
jobLog.setTriggerMsg(triggerMsgSb.toString());
XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().updateTriggerInfo(jobLog);
/// RedisLockUtil.releaseLock(jobInfo.getId()+"",uid);
logger.debug(">>>>>>>>>>> xxl-job trigger end, jobId:{}", jobLog.getId());
}
/**
* run executor
*
* @param triggerParam
* @param address
* @return
*/
public static ReturnT<String> runExecutor(ExpandTriggerParam triggerParam, String address) {
ReturnT<String> runResult = null;
try {
ExecutorBiz executorBiz = XxlJobScheduler.getExecutorBiz(address);
runResult = executorBiz.run(triggerParam);
} catch (Exception e) {
logger.error(">>>>>>>>>>> xxl-job trigger error, please check if the executor[{}] is running.", address, e);
runResult = new ReturnT<String>(ReturnT.FAIL_CODE, ThrowableUtil.toString(e));
}
StringBuffer runResultSB = new StringBuffer(I18nUtil.getString("jobconf_trigger_run") + ": ");
runResultSB.append("<br>address: ").append(address);
runResultSB.append("<br>code: ").append(runResult.getCode());
runResultSB.append("<br>msg: ").append(runResult.getMsg());
runResult.setMsg(runResultSB.toString());
return runResult;
}
}