Background:

• Project uses Python's own logging library to print logs
• The project is deployed on a Centos7 machine
• Project uses gunicorn multi-process deployment

Process:

1. LOG Log Code Encapsulation:

Use the logging library and set when='MIDNIGHT', split the log in days, the log of the previous day will be automatically added with the date of the previous day, the latest log will always be printed to the mock-service.log file, the following is the encapsulation of log printing

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Time    : 2019/8/8 22:10
# @Author  : 
# @Site    : 
# @File    : Logger.py
# @Software: PyCharm

"""
//Log class.Define the log level, log file name, log format, and so on by reading the configuration file.
//Normally log ger import is entered directly
from utils.log import logger
logger.info('test log')
"""
import os
import platform
import logging
from logging.handlers import TimedRotatingFileHandler


class Logger:
    def __init__(self, logger_name='framework'):
        self.logger = logging.getLogger(logger_name)
        logging.root.setLevel(logging.NOTSET)
        if platform.system() == 'Windows':
            # win Machine Path
            self.log_path=os.path.join(os.path.dirname(os.path.dirname(__file__)),'log')
        else:
            # Server Path
            self.log_path='/log'
        self.log_file_name = 'mock-service.log'  # log file
        self.backup_count = 30  # Number of logs to keep
        # Log Output Level
        self.console_output_level = 'INFO'
        self.file_output_level = 'INFO'
        # Log Output Format
        pattern = '%(asctime)s - %(filename)s [Line:%(lineno)d] - %(levelname)s - %(message)s'
        self.formatter = logging.Formatter(pattern)

    def get_logger(self):
        """stay logger Add a log handle to and return if logger If a handle already exists, return directly
        //Here we add two handles, one to output the log to the console and the other to the log file.
        //The log levels of the two handles are different and can be set in the configuration file.
        """
        if not self.logger.handlers:  # Avoid duplicate logs
            console_handler = logging.StreamHandler()
            console_handler.setFormatter(self.formatter)
            console_handler.setLevel(self.console_output_level)
            self.logger.addHandler(console_handler)

            # Create a new log file every day, keeping up to backup_count copies
            # The log file generated each day has no suffix and the source code needs to be modified: the doRollover method under the TimedRotatingFileHandler class-->
            # DFN = self.rotation_filename (self.baseFilename +'."+ time.strftime (self.suffix, timeTuple) followed by a stitched suffix name
            # dfn = self.rotation_filename(self.baseFilename + "." + time.strftime(self.suffix, timeTuple) + ".log")
            file_handler = TimedRotatingFileHandler(filename=os.path.join(self.log_path, self.log_file_name),
                                                    when='MIDNIGHT',
                                                    interval=1,
                                                    backupCount=self.backup_count,
                                                    delay=True,
                                                    encoding='utf-8'
                                                    )
            file_handler.setFormatter(self.formatter)
            file_handler.setLevel(self.file_output_level)
            self.logger.addHandler(file_handler)
        return self.logger

logger = Logger().get_logger()

2. Call LOG encapsulation

When actually invoked, log printing is done by importing the package and setting an alias. Here is the demo of the actual log printed

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Time    : 2019/12/2 10:32
# @Author  : 
# @Site    : 
# @File    : 1.py
# @Software: PyCharm
# @Description:

from util.Logger import logger as log

class test1:
    def log_test(self):
        log.info('log test')
    
    
    
if __name__ == '__main__':
    t = test1()
    t.log_test()

3. Project Deployment Environment

The project is currently deployed on a server in centos7, but I don't think it matters where it is deployed.

4. Problems (pits, pits)

Projects continue to run on the server, but sometimes you find that the logs are not printed at all to the mock-service.log file, such as the following, the call log of December 29, reached the log file of No. 27

Solution ideas:

1. First attempt:

To find out why, some of the previous calls to log printing were written in the init method, then the following method was called with self.log. Later, to solve this problem, all init initializations were removed. Now I think the problem may still be on the call, and the log packaging should be OK.Now some of the class methods in this project are printing logs and some of the encapsulated functions are printing logs.The whole project was written in the flask framework, and then the run.py file was started. I think maybe the call that happened somewhere took up the process of log printing, causing the later call to log printing to print to the previous file. It's just a guess, and I don't know how to verify this problem.

2. Second attempt

After another Baidu, Google search, found that Python's own logging library does not support multi-process, later found that this problem can be solved by the following way

• Rewrite the logging library yourself to fix this problem
• Through third-party libraries already encapsulated by others

I chose the second option, not repeating the wheel, and after a comparison, I finally chose this third-party library, concurrent_log, which can be viewed in detail in the github repository, and the author tested the rewritten code to prove that it solved the multithreaded, multiprocess problem.

Installation method:

pip3 install concurrent_log

github address:

https://github.com/huanghyw/concurrent_log

It's also possible to override the logging library yourself, so here are a few overridden articles

• https://www.cnblogs.com/restran/p/4743840.html
• https://www.jianshu.com/p/d874a05edf19