Log is a good thing to locate historical problems, but there are too many records, do not scroll, and do not remove the old total disk burst. If the logs are output by the log framework, such as Log4j or Logback, you can scroll the logs and delete the old archived log files by selecting an Appender with scrolling feature. However, there are log output files that are difficult to control in the program. If this is used, remedial measures must be taken afterwards. Although the program writes to one log file, another program archives and cleans the log file.

We can find the following tools related to this

1. logrotate, which comes with most Linux distributions today, feels like a home advantage. On github
   logrotate/logrotate is still active

2. The newsyslog, FreeBSD, and MAC systems come with them and should not be used frequently. You can view the configuration file under Mac OS
   /etc/newsyslog.conf

3. Cronolog, the original official website www.cronolog.org is all in Japanese. Find its snapshot fleible web log
   The latest update of fordmason/cronolog on rotation and GitHub was five years ago

4. rotatelogs, from the Apache HTTP project, is used by the Apache HTTP server to scroll access and error logs

I highly recommend using the first tool logrotate, because most Linux systems come with it, unlike cronolog and rotatelogs
Additional installation is required. It also has more complete functions. Let's enjoy it slowly

Working mechanism of logrotate

Under Linux, there is a Cron Job executed every day by default, which is configured in / etc/cron.daily/logrotate, and the file content is (take centos7 as an example)

#!/bin/sh
 
/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

The above means that Linux uses the configuration file / etc/logrotate.conf to execute the command logrotate every day, and the execution status is written in the / var/lib/logrotate/logrotate.status file. We can view the status log file to confirm the actual behavior of logrotate.

The default execution interval of logrotate itself is once a day, so even if hourly is used in our own configuration, it is useless unless we move the above logrotate from / etc/cron.daily to / etc/cron.hourly directory.

The / etc/cron.daily/logrotate in Ubuntu is similar. In short, the / etc/logrotate.conf configuration file should be applied.

The main contents of the configuration file are as follows:

#The following five lines are the global default configuration for log scrolling
weekly #The default is one log archive per week
rotate 4 #Save up to 4 archives
create #Create a new log file after log scrolling
dateext #Archive file name with date suffix
#compress #Whether archive file compression is enabled
 
# Contains all configuration files in the / etc/logrotate.d / directory
include /etc/logrotate.d
 
# This is an example of specifying a log file archive configuration
/var/log/wtmp {
    monthly
    create 0664 root utmp
	minsize 1M
    rotate 1
}

Therefore, we hope to automatically archive a log file. The configuration can be written directly in / etc/logrotate.conf file. Like the section of / var/log/wtmp {... You can also create a separate configuration file in the directory / etc/logrotate.conf. The latter is strongly recommended.

Create your own logrotate configuration

For configuration files, please refer to several practical examples in the / etc/logrotate.d directory. You can see configuration files such as bootlog, Chrony, syslog, wpa_supplier, yum under Centos7. For complete configuration instructions, please refer to logrotate(8) - Linux man page.

If we need to scroll the access log of httpd, we can create a file httpd_access_log in the / etc/logrotate.d/ directory and put the contents on the

/var/log/httpd/access.log {
    rotate 5
    size 20M
    compress
    copytruncate
    dateext
    sharedscripts
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
}

The above configuration can be archived every day if the log file / var/log/httpd/access.log reaches 20m

/var/log/httpd/access-20180601.gz

Clear the original log file / var/log/httpd/access.log, and send a HUP signal to the httpd process.

At most 5 archive log files are reserved. If there is no dateext configuration item above, the generated archive files will be file names such as access.1.gz and access.2.gz.

For more configuration options in the configuration file, please refer to logrotate(8) - Linux man page , its options are not specifically explained here, but only a simple example of what can be achieved with the help of configuration items

• You can send mail when archiving
• You can set the date format in the archive suffix
• Archive logs can be stored in other directories. They exist in the same directory by default
• Custom scripts can be executed before and after archiving
• Matching multiple log files can specify whether the script is triggered for each log file or only once

After the file / etc/logrotate.d/httpd_access_log is created, you can lie there quietly without any service restart operation. You can see the results the next day. If you can't wait, use the date command to modify the system time. Be careful not to cause confusion in the running program date.

Debug configuration file

To use the logrotate command, you can logrotate /etc/logrotate.conf, which will trigger all log scrolling operations. It should not be what we want, so it should be simple

logrotate -d -f /etc/logrotate.d/httpd_access_log

Add a - d (– debug), which is equivalent to dry running. You can see the following analog operation output

$ logrotate -d -f /etc/logrotate.d/httpd_access_log
reading config file /etc/logrotate.d/httpd_access_log
Allocating hash table for state file, size 15360 B

Handling 1 logs

rotating pattern: /var/log/httpd/access.log forced from command line (5 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/httpd/access.log
    log needs rotating
rotating log /var/log/httpd/access.log, log->rotateCount is 5
dateext suffix '-20211001'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
glob finding old rotated logs failed
copying /var/log/httpd/access.log to /var/log/httpd/access.log-20180601
truncating /var/log/httpd/access.log
compressing log with: /bin/gzip

If you want to see its actual effect, remove - d from it

$ logrotate -f /etc/logrotate.d/httpd_access_log

Note that a - f (– force) is added in front of the log file to force log scrolling regardless of whether the log file size meets the requirements

To be more aggressive, you can also modify the system time to trigger the execution of logrotate in Cron Job. For example, today is 2021-10-02, which is changed to No. 2

$ sudo date -s 2021-10-02

After a while, you can check whether the log file has scrolled or the logrotate status log / var/lib/logrotate/logrotate.status.

Multiple log files and wildcards

A configuration entry log file {configuration item} supports more than one log file. You can configure multiple files or use wildcards, such as

/var/log/httpd/access.log /var/log/httpd/error.log {
....
}
 
# or
/var/log/httpd/access.log
/var/log/httpd/error.log {
...
}

Wildcard form

/var/log/news/* {
...
}
 
/var/log/news/*.log {
...
}
 
/var/log/*/stdout.log {
...
}
 
/var/log/*/*.log {
...
}

Wildcards can be used not only in the file name, but also in the directory. For example, the last configuration / var/log//.log {...} will have an effect on *. Log files in all directories under / var/log /. For example, the following files

/var/log/aa/x.log
/var/log/bb/y.log
/var/log/cc/z.log

For multiple log files, the archive files will also be generated in the corresponding log directory. With the support of multiple log files and wildcards, you can take consistent actions on many log files in the system through one configuration.

If you use > to redirect the output log, such as test_app > stdout.log and copyruncat, the original log file stdout.log cannot be cropped, and it will continue to grow. If you use > >, there is no problem. Test app > > stdout.log, and the size of stdout.log will become zero after the log scrolls.