Database Archiving Strategy

Keywords: Database SQL

Database migration strategy

In order to prepare for Shuang11, the relevant tables (historical orders) in the database need to be filed in order to make more space for the surge of orders. The author tries to find the best solution. The following is the database archiving strategy and sample code.
Existing conditions:
1. There are two existing databases: db-A and db-B.
2. Tables with the same field in the two libraries: tba (only field order id-rx_id (long type) has an index in the table);
3. There are 17 years of filing data in the tba of the archive library.
4. Because of the concealment of single quantity, rx_id does not increase in turn according to the next single time, but the general trend is increasing. Namely: Make sure that today's maximum number is larger than yesterday's maximum number.
5. About 300w units need to be migrated.

We note that,
1. Because of the huge amount of data, the best way to query sql is to use index.
2. There is only one index available to the database.

Solutions:
1. First of all, the author gets the largest order number in the Danggui library table: rx_id_start.
2. Second, get the largest order number on the day of the closing time: rx_id_end.
3. Query the orders in the range of two order numbers by page through two order numbers.
4. Insert the archive table in batches according to the number of pages.
5. Make a regular job and execute it regularly.

Benefits:
1. Maximize and skillfully use the index, which enhances the retrieval speed.
2. It can be made into a regular job once a day, and the filing work is sustainable.

Complete code:

private static String archive4RxInfoJob = "archive4RxInfoJob";
 /**
     * Maximum rxId of cache end time
     */
public static Map<String, Long> currentMaxRxIdMap = new ConcurrentHashMap<>();
//Packaged thread pool
private ThreadPoolService threadPoolService;
private static int pageSize = 100;

//Timely job execution
public JsfResult<Boolean> archive4RxInfoJob() {
        final Long start = System.currentTimeMillis();
        JsfResult<Boolean> result = new JsfResult<>();
        //Start ump monitoring
        CallerInfo umpCall = umpUtil.start("medicine-b2c-man.method.DataArchiveExportServiceImpl.archive4RxInfoOneTime");
        try {
            final Date endTm = getEndDay();
            //Asynchronous execution
            this.threadPoolService.asynExecuteTask(new Runnable() {
                @Override
                public void run() {
                    int i = 0;
                    while (true) {
                        if (i == 10) {
                            break;
                        }
                        //Get the largest rx_id in the archive in turn
                        Long rxId = dataArchiveDao.getRxInfoByRxIdAndEndTmPage();
                        //Get the rx_id of the deadline in the production repository table and record it in the local cache.
                        Long maxRxIdEndTm = currentMaxRxIdMap.get(archive4RxInfoJob + endTm.getTime());
                        if (maxRxIdEndTm == null) {
                            maxRxIdEndTm = rxReadDao.getMaxRxIdEndDay(endTm);
                            currentMaxRxIdMap.put(archive4RxInfoJob + endTm.getTime(), maxRxIdEndTm);
                        }
                        //Query 1000 pieces of data
                        logger.info("archive4RxInfoJob->start->rxId:{},date:{}", rxId, DateUtil.formatDate(endTm, "yyyy-MM-dd HH:mm:ss"));
                        List<RxInfo> resList = rxReadDao.getRxInfoByRxIdPage(rxId, pageSize / 10, maxRxIdEndTm);
                        if (resList == null || resList.size() == 0) {
                            logger.info("archive4RxInfoJob->resList size is 0");
                            return;
                        }
                        //Batch insertion
                        dataArchiveDao.batchAddRxInfoArchive(resList );
                        logger.info("archive4RxInfoJob->after->rxId:{},date:{},last:{}", rxId, DateUtil.formatDate(endTm, "yyyy-MM-dd HH:mm:ss"), System.currentTimeMillis() - start);
                        i++;
                    }
                }
            });

            result.setMsg(DataArchiveErrorCode.SUCCESS.getDescription());
            result.setCode(DataArchiveErrorCode.SUCCESS.getCode());
            return result;
        } catch (Exception e) {
            result.setCode(BusinessErrorCode.UNKNOWN_ERROR.getCode());
            result.setMsg(BusinessErrorCode.UNKNOWN_ERROR.getDescription());
            return result;
        } 
    }
    private static Date getEndDay() throws Exception {
        //Determine the time range, only guide the data half a year ago
        long current = System.currentTimeMillis();//Current time milliseconds
        long zero = current / (1000 * 3600 * 24) * (1000 * 3600 * 24) - TimeZone.getDefault().getRawOffset();//Number of milliseconds in zero minutes and zero seconds today
        Date endTm = DateUtil.addDays(DateUtil.parseDate(new Timestamp(zero).toString(), "yyyy-MM-dd HH:mm:ss.S"), -180);
        return endTm;
    }

Posted by Atomic Taco on Wed, 30 Jan 2019 00:15:15 -0800