org.quartz implements timed tasks and customizes task data source switching

Timing tasks are often used to deal with periodic tasks. org.quartz is an excellent framework for dealing with such tasks. As the project progresses little by little, we are not satisfied that the task is only carried out on time. We also want to have more control over the task. If we can intervene in the task at any time, we need to have a deeper understanding of quartz. With the popularity of micro services, the situation of multi-data sources in projects is becoming more and more common. The function of integrating multi-data source switching in timing tasks also needs to be integrated.

Integrating quartz to implement timing tasks

Integrating quartz to implement timing tasks

Basic concepts needed to be understood in quartz to implement timing tasks

Job

By implementing the Job class, we write the specific tasks we want to accomplish in the implementation method, and then give them to quartz management.

JobDetail

Job is only responsible for specific tasks, so we need to use JobDetail to store some basic information describing Job.

Quartz JobBuilder

builder-style API for constructing JobDetail entities. You can use it to build a JobDetail in this way:

@Bean
public JobDetail jobDetail() {
 return JobBuilder.newJob().ofType(SampleJob.class)
 .storeDurably()
 .withIdentity("Qrtz_Job_Detail")
 .withDescription("Invoke Sample Job service...")
 .build();
}

Spring JobDetailFactoryBean

How to configure JobDetail in Spring:

@Bean
public JobDetailFactoryBean jobDetail() {
 JobDetailFactoryBean jobDetailFactory = new JobDetailFactoryBean();
 jobDetailFactory.setJobClass(SampleJob.class);
 jobDetailFactory.setDescription("Invoke Sample Job service...");
 jobDetailFactory.setDurability(true);
 return jobDetailFactory;
}

Trigger

Triggers, representing the configuration of a scheduling parameter, when to schedule:

@Bean
public Trigger trigger(JobDetail job) {
 return TriggerBuilder.newTrigger().forJob(job)
 .withIdentity("Qrtz_Trigger")
 .withDescription("Sample trigger")
 .withSchedule(simpleSchedule().repeatForever().withIntervalInHours(1))
 .build();
}

Scheduler

Scheduler registers a scheduler through Job and Trigger:

@Bean
public Scheduler scheduler(Trigger trigger, JobDetail job) {
 StdSchedulerFactory factory = new StdSchedulerFactory();
 factory.initialize(new ClassPathResource("quartz.properties").getInputStream());

 Scheduler scheduler = factory.getScheduler();
 scheduler.setJobFactory(springBeanJobFactory());
 scheduler.scheduleJob(job, trigger);

 scheduler.start();
 return scheduler;
}

Add a Job to the system

In quartz, Job is the task we need to perform. Scheduler scheduler is responsible for dispatching tasks, and they depend on Trigger to perform tasks regularly.

So first we need to add a Job to the system based on the above.

addJob

    public void addJob(BaseJob job) throws SchedulerException {
        /** Create an instance of JobDetail and bind the Job implementation class
        * JobDetail Represents a specific executable scheduler. job is the content of the executable scheduler to execute.
        * In addition, JobDetail also includes the schemes and strategies for task scheduling.**/
        // Specify the name of the job, the name of the group in which it belongs, and the bound job class
        JobDetail jobDetail = JobBuilder.newJob(job.getBeanClass())
                .withIdentity(job.getJobKey())
                .withDescription(job.getDescription())
                .usingJobData(job.getDataMap())
                .build();

        /**
         * Trigger Represents the configuration of a scheduling parameter, when to schedule
         */
        //Define scheduling trigger rules, using cronTrigger rules
        Trigger trigger = TriggerBuilder.newTrigger()
                .withIdentity(job.getJobName(),job.getJobGroup())
                .withSchedule(CronScheduleBuilder.cronSchedule(job.getCronExpression()))
                .startNow()
                .build();
        //Register tasks and triggers into task scheduling
        scheduler.scheduleJob(jobDetail,trigger);
        //Determine whether the scheduler is started
        if(!scheduler.isStarted()){
            scheduler.start();
        }
        log.info(String.format("Timing tasks:%s.%s-Added to Scheduler!", job.getJobGroup(),job.getJobName()));
    }

First we need to define our Job, then initialize JobDetail and Trigger through Job, and finally register JobDetail and Trigger in the scheduler.

BaseJob

Job is structured as follows:

public abstract class BaseJob implements Job,Serializable {
    private static final long serialVersionUID = 1L;
    private static final String JOB_MAP_KEY = "self";
    /**
     * Task Name
     */
    private String jobName;
    /**
     * Task grouping
     */
    private String jobGroup;
    /**
     * Whether the task state starts the task or not
     */
    private String jobStatus;
    /**
     * cron Expression
     */
    private String cronExpression;
    /**
     * describe
     */
    private String description;
    /**
     * Which class's method package name + class name is called during task execution
     */
    private Class beanClass = this.getClass();
    /**
     * Is the task stateful?
     */
    private String isConcurrent;

    /**
     * Spring bean
     */
    private String springBean;

    /**
     * Method name for task invocation
     */
    private String methodName;

     /**
     * Data sources used for this task
     */
    private String dataSource = DataSourceEnum.DB1.getName();

    /**
     * To persist the executed task to the database
     */
    @JsonIgnore
    private JobDataMap dataMap = new JobDataMap();

    public JobKey getJobKey(){
        return JobKey.jobKey(jobName, jobGroup);// Task Name and Composition Task key
    }
    ...
}

You can see some basic information about tasks defined in Job, focusing on the dataSource and dataMap attributes. The dataSource is the data source used by the task and gives a default value; because the task is added and persisted to the database, then the task is parsed using the dataMap.

SchedulerConfig

When Job is added, JobDetail and Trigger are generated by keyword new, while Scheduler needs to be maintained in containers.

@Configuration
@Order
public class SchedulerConfig {
    @Autowired
    private MyJobFactory myJobFactory;

    @Value("${spring.profiles.active}")
    private String profile;

    /*
     * Get an instance of Scheduler through Scheduler FactoryBean
     */
    @Bean(name = "scheduler")
    public Scheduler scheduler() throws Exception {
        return schedulerFactoryBean().getScheduler();
    }
    
    @Bean
    public SchedulerFactoryBean schedulerFactoryBean() throws IOException {
        SchedulerFactoryBean factory = new SchedulerFactoryBean();

        factory.setOverwriteExistingJobs(true);

        // Delayed startup
        factory.setStartupDelay(20);

        // Loading quartz data source configuration
        factory.setQuartzProperties(quartzProperties());

        // Custom Job Factory for Spring Injection
        factory.setJobFactory(myJobFactory);
        /*********Global listener configuration******************/
        JobListener myJobListener = new SchedulerListener();
        factory.setGlobalJobListeners(myJobListener);//Add directly to the global listener
        return factory;
    }

    @Bean
    public Properties quartzProperties() throws IOException {
        PropertiesFactoryBean propertiesFactoryBean = new PropertiesFactoryBean();
        if (Util.PRODUCT.equals(profile)) {//Formal environment
            System.out.println("Formal environment quartz To configure");
            propertiesFactoryBean.setLocation(new ClassPathResource("/quartz-prod.properties"));
        } else {
            System.out.println("testing environment quartz To configure");
            propertiesFactoryBean.setLocation(new ClassPathResource("/quartz.properties"));
        }
        //Properties in quartz.properties are read and injected before the object is initialized
        propertiesFactoryBean.afterPropertiesSet();
        return propertiesFactoryBean.getObject();
    }

    /*
     * quartz Initialization listener
     */
    @Bean
    public QuartzInitializerListener executorListener() {
        return new QuartzInitializerListener();
    }
}

In the above code, add scheduler to the Spring container. Scheduler is maintained by Scheduler FactoryBean. Scheduler FactoryBean provides some basic settings for the Scheduler Factory and loads the quartz data source configuration from the configuration file (the reading of the configuration file is automatically switched according to the running environment profile). A global listener is configured to listen for tasks. Implementation process.

MyJobFactory

Use the JobFactory provided by Spring.

@Component
public class MyJobFactory extends AdaptableJobFactory {

    @Autowired
    private AutowireCapableBeanFactory capableBeanFactory;

    @Override
    protected Object createJobInstance(TriggerFiredBundle bundle) throws Exception {
        // Calling methods of parent classes
        Object jobInstance = super.createJobInstance(bundle);
        // Injection
        capableBeanFactory.autowireBean(jobInstance);
        return jobInstance;
    }
}

quartz.properties

Quatz. properties is some configuration information for quartz to connect to the database.

# \u56FA\u5B9A\u524D\u7F00org.quartz
# \u4E3B\u8981\u5206\u4E3Ascheduler\u3001threadPool\u3001jobStore\u3001plugin\u7B49\u90E8\u5206
#
#
org.quartz.scheduler.instanceName = DefaultQuartzScheduler
org.quartz.scheduler.rmi.export = false
org.quartz.scheduler.rmi.proxy = false
org.quartz.scheduler.wrapJobExecutionInUserTransaction = false

# \u5B9E\u4F8B\u5316ThreadPool\u65F6\uFF0C\u4F7F\u7528\u7684\u7EBF\u7A0B\u7C7B\u4E3ASimpleThreadPool
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool

# threadCount\u548CthreadPriority\u5C06\u4EE5setter\u7684\u5F62\u5F0F\u6CE8\u5165ThreadPool\u5B9E\u4F8B
# \u5E76\u53D1\u4E2A\u6570
org.quartz.threadPool.threadCount = 5
# \u4F18\u5148\u7EA7
org.quartz.threadPool.threadPriority = 5
org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread = true

org.quartz.jobStore.misfireThreshold = 5000

# \u9ED8\u8BA4\u5B58\u50A8\u5728\u5185\u5B58\u4E2D
#org.quartz.jobStore.class = org.quartz.simpl.RAMJobStore

#\u6301\u4E45\u5316
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX

#org.quartz.jobStore.useProperties=false

org.quartz.jobStore.tablePrefix = QRTZ_

org.quartz.jobStore.dataSource = qzDS

org.quartz.dataSource.qzDS.driver = com.mysql.jdbc.Driver
org.quartz.dataSource.qzDS.URL=jdbc:mysql://127.0.0.1:3306/quartz?characterEncoding=UTF-8&useSSL=false&testOnBorrow=true&testWhileIdle=true
org.quartz.dataSource.qzDS.user=quartz
org.quartz.dataSource.qzDS.password=123456

org.quartz.dataSource.qzDS.maxConnections = 30

org.quartz.dataSource.qzDS.validationQuery = SELECT 1 FROM DUAL

org.quartz.dataSource.qzDS.validateOnCheckout = true
org.quartz.dataSource.qzDS.idleConnectionValidationSeconds = 40


#org.quartz.dataSource.qzDS.discardIdleConnectionsSeconds = 60

Quartz persists Job to the database based on this configuration file, so quartz will need to initialize some database tables and table structure files at the end.

SchedulerListener

Scheduler listeners are used to monitor the execution status of tasks.

public class SchedulerListener implements JobListener {

    private final Logger LOG = LoggerFactory.getLogger(SchedulerListener.class);

    public static final String LISTENER_NAME = "QuartSchedulerListener";

    @Override
    public String getName() {
        return LISTENER_NAME; //must return a name
    }

    //Before the task is scheduled
    @Override
    public void jobToBeExecuted(JobExecutionContext context) {
        String dataSource = context.getJobDetail().getJobDataMap().getString("dataSource");
        // Data Source for Switching Tasks
        DataSourceContextHolder.setDB(dataSource);
        String jobName = context.getJobDetail().getKey().toString();
        LOG.info("Job {} is going to start，switch dataSource to {}，Thread name {}", jobName, dataSource, Thread.currentThread().getName());
    }

    //Task scheduling was rejected
    @Override
    public void jobExecutionVetoed(JobExecutionContext context) {
        String jobName = context.getJobDetail().getKey().toString();
        LOG.error("job {} is jobExecutionVetoed", jobName);
        //You can do some logging for reasons

    }

    //After the task is scheduled
    @Override
    public void jobWasExecuted(JobExecutionContext context,
                               JobExecutionException jobException) {
        // Empty the stored data source
        String jobName = context.getJobDetail().getKey().toString();
        DataSourceContextHolder.clearDB();
        LOG.info("Job : {} is finished", jobName);
        if (jobException != null && !jobException.getMessage().equals("")) {
            LOG.error("Exception thrown by: " + jobName
                    + " Exception: " + jobException.getMessage());
        }

    }
}

SchedulerListener monitors the status of tasks before, after and when they are rejected, and processes the data sources used by tasks before and after they are scheduled. If there is no need for data source switching in the project, the listener is not needed, and quartz integration has been completed.

Multiple Data Source Switching

Multiple Data Source Switching

Overlay the original data source in Spring Boot by customizing DynamicDataSource.

DataSourceConfig

By reading different data sources in the configuration file, the data sources that may be used in the initialization project are initialized for switching.

/**
 * Multiple Data Source Configuration Class
 */
@Configuration
public class DataSourceConfig {
    //Data Source 1
    @Bean(name = "datasource1")
    @ConfigurationProperties(prefix = "spring.datasource.db1") // Prefixes for corresponding attributes in application. properties
    public DataSource dataSource1() {
        return DataSourceBuilder.create().build();
    }

    //Data Source 2
    @Bean(name = "datasource2")
    @ConfigurationProperties(prefix = "spring.datasource.db2") // Prefixes for corresponding attributes in application. properties
    public DataSource dataSource2() {
        return DataSourceBuilder.create().build();
    }

    /**
     * Dynamic Data Sources: Dynamic Switching between Different Data Sources through AOP
     *
     * @return
     */
    @Primary
    @Bean(name = "dynamicDataSource")
    public DataSource dynamicDataSource() {
        DynamicDataSource dynamicDataSource = new DynamicDataSource();
        // Default data source
        dynamicDataSource.setDefaultTargetDataSource(dataSource1());
        // Configure multiple data sources
        Map<Object, Object> dsMap = new HashMap();
        dsMap.put(DataSourceEnum.DB1.getName(), dataSource1());
        dsMap.put(DataSourceEnum.DB2.getName(), dataSource2());

        dynamicDataSource.setTargetDataSources(dsMap);
        return dynamicDataSource;
    }

    @Bean
    public SqlSessionFactory sqlSessionFactory(DataSource dataSource) throws Exception {
        SqlSessionFactoryBean sqlSessionFactoryBean = new SqlSessionFactoryBean();
        //set up data sources
        sqlSessionFactoryBean.setDataSource(dataSource);
        return sqlSessionFactoryBean.getObject();
    }

    /**
     * Configure @Transactional annotations
     *
     * @return
     */
    @Bean
    public PlatformTransactionManager transactionManager() {
        return new DataSourceTransactionManager(dynamicDataSource());
    }
}

Data source configuration

spring:
  datasource:
    db1:
      driver-class-name: com.mysql.cj.jdbc.Driver
      username: doctor
      password: 123456
      type: com.zaxxer.hikari.HikariDataSource
      jdbc-url: jdbc:mysql://127.0.0.1:3306/doctor?useSSL=false&testOnBorrow=true&testWhileIdle=true
    db2:
      driver-class-name: com.mysql.cj.jdbc.Driver
      username: quartz
      password: 123456
      type: com.zaxxer.hikari.HikariDataSource
      jdbc-url: jdbc:mysql://127.0.0.1:3307/quartz?useSSL=false&testOnBorrow=true&testWhileIdle=true

DataSourceContextHolder

Because quartz executes Job through different threads during execution, ThreadLocal is used here to save the data source used by threads.

/**
 * Save local data sources
 */
public class DataSourceContextHolder {
    private static final Logger LOG = LoggerFactory.getLogger(DataSourceContextHolder.class);
    /**
     * Default data source
     */
    public static final String DEFAULT_DS = DataSourceEnum.DB1.getName();
    /**
     * ThreadLocal It will be explained later.
     */
    private static final ThreadLocal<String> contextHolder = new ThreadLocal<>();

    // Setting the name of the data source
    public static void setDB(String dbType) {
        LOG.info("switch to{}data source", dbType);
        contextHolder.set(dbType);
    }

    // Get the data source name
    public static String getDB() {
        return (contextHolder.get());
    }

    // Clear the data source name
    public static void clearDB() {
        contextHolder.remove();
    }
}

DynamicDataSource

Gets the data source used in the execution. Since the data source is stored in ThreadLocal in the Data Source ContextHolder, it's all right to get it directly.

/**
 * Getting local data sources
 */
public class DynamicDataSource extends AbstractRoutingDataSource {
    private static final Logger LOG = LoggerFactory.getLogger(DynamicDataSource.class);

    @Override
    protected Object determineCurrentLookupKey() {
        LOG.info("The data source is{}", DataSourceContextHolder.getDB());
        return DataSourceContextHolder.getDB();
    }
}

So far, the function of integrating quartz and data source switching has been completed. Then there is the specific task.

Execution of tasks

Specific tasks need to inherit BaseJob and rewrite specific tasks in the execute method.

execute

@Slf4j
@Service
public class ReadNumJob extends BaseJob {

    @Autowired
    private RedisService redisService;

    @Autowired
    private JdbcTemplate jdbcTemplate;

    private final Logger LOG = LoggerFactory.getLogger(ReadNumJob.class);

    @Override
    public void execute(JobExecutionContext context) {
       doSomething();
    }
}

specify data source

Then specify the data source used by the task when adding the task

ReadNumJob job = new ReadNumJob();
job.setJobName("test");
job.setJobGroup("hys");
job.setDescription("test");
// specify data source
job.getDataMap().put("dataSource", DataSourceEnum.DB1.getName());
job.setCronExpression(
"0 */1 * * * ?"
);
try {
jobAndTriggerService.addJob(job);
} catch (SchedulerException e) {
e.printStackTrace();
}

Source code

Re-evaluation is the greatest encouragement