Log segmentation
-
As the number of visits to the website increases, Apache's individual log files will grow by default
- Log files take up a lot of disk space
- Inconvenient to view relevant information
- Split log files
- Implementation of rotatelogs segmentation tool in Apache
- Third party tool cronolog segmentation
rotatelogs segmentation
- The log files of the configuration website are transferred to rotatelogs for split processing
- The configuration format is
ErrorLog 4| rotatelogs Absolute path of command-I log file path/Website name-error %Y%m%d.log 86400"- CustomLog "| rotatelogs Command path-I log file path/Website name-access %Y%m%d.log 86400" combined
Coordination example
[root@localhost ~]# yum install httpd -y / / install http service
//Loaded plug-ins: faststmirror, langpacks
Loading mirror speeds from cached hostfile
* base: centos.ustc.edu.cn
* extras: centos.ustc.edu.cn
...//Omit parts
//Installed:
httpd.x86_64 0:2.4.6-90.el7.centos
...//Omit parts
//Complete!
[root@localhost bin]# cd /usr/sbin / / / enter sbin directory
[root@localhost sbin]# Ls rotate * / / check whether the command is generated in the directory
rotatelogs //Generate rotatelogs command after installing http service
[root@localhost sbin]# cd /var/log/httpd / / / enter the http service log directory
[root@localhost httpd]# ls / / check whether log files are generated
[root@localhost httpd]# //No catalog file generated after installation
[root@localhost httpd]# vim /etc/httpd/conf/httpd.conf / / enter the http configuration file and perform general configuration
...//Omit parts
# prevent Apache from glomming onto all bound IP addresses.
#
Listen 192.168.144.133:80 //Modify IPv4 to local address
#Listen 80 / / comment out IPv6 listening
...//Omit parts
# If your host doesn't have a registered DNS name, enter its IP address here.
#
ServerName www.kgc.com:80 //Open domain name service
#
# Deny access to the entirety of your server's filesystem. You must
...//Omit parts
:wq
[root@localhost httpd]# systemctl stop firewalld.service / / turn off the firewall
[root@localhost httpd]# setenforce 0 / / turn off enhanced security
[root@localhost httpd]# systemctl start httpd.service / / start http service
[root@localhost httpd]# ls / / check whether log files are generated
access_log error_log //Log file generated successfully after service startup
[root@localhost httpd]# vim conf/httpd.conf / / enter and change the main configuration file to do log splitting
...//Omit parts
# container, that host's errors will be logged there and not here.
#
#Error log "logs / error_log" / / copy this entry and paste it below
ErrorLog "| /usr/sbin/rotatelogs -l logs/www.kgc.com.error_%Y%m%log 86400"
//Change the entry, reference the rotatelogs command, create the split log entry name, and create the file every 86400 seconds (one day)
# LogLevel: Control the number of messages logged to the error_log.
...//Omit parts
# (Combined Logfile Format) you can use the following directive.
#
#CustomLog "logs / access? Log" combined / / copy this entry and paste it below
CustomLog "| /usr/sbin/rotatelogs -l logs/www.kgc.com.access_%Y%m%dlog 86400" combined
//Change the entry, reference the rotatelogs command, create the split log entry name, and create the file every 86400 seconds (one day)
</IfModule>
<IfModule alias_module>
...//Omit parts
:wq
[root@localhost httpd]# systemctl stop httpd.service / / stop http service
[root@localhost httpd]# systemctl start httpd.service / / start http service
[root@localhost httpd]# cd ..
[root@localhost log]# cd ~
[root@localhost ~]# cd /var/log/httpd / / / re enter the site file
[root@localhost httpd]# ls
access_log error_log www.kgc.com.error_20191105log //Show split log files
[root@localhost httpd]# date / / view time
2019 Tuesday, November 5, 19:27:50 CST
[root@localhost httpd]# date -s 11/06/19 / / the day after setting the time
2019 Wednesday, November 6, 2006:00:00 CST
[root@localhost httpd]# systemctl stop httpd.service / / close http service
[root@localhost httpd]# systemctl start httpd.service / / start the service
[root@localhost httpd]# cd ~ / / home directory
[root@localhost ~]# cd /var/log/httpd / / / restore today's log directory
[root@localhost httpd]# ls / / view
access_log error_log www.kgc.com.error_20191105log www.kgc.com.error_20191106log
//Generate a new log fileThird party tool cronolog
- Source code compilation and installation cronolog tool cronolog installation package download
- Configure website log file to be transferred to cronolog for split processing
- The configuration format is:
ErrorLog "| cronolog Absolute path to command log file path/Website name-error_ %Y%m%d.log' CustomLog "| cronolog Absolute path of command|log file path/Website name%Y%m%d.log" combined
Configuration example
Restart a Linux virtual machine and reinstall the http service
[root@localhost ~]# yum install httpd -y / / install the service
//Loaded plug-ins: faststmirror, langpacks
Loading mirror speeds from cached hostfile
* base: centos.ustc.edu.cn
...//Omit parts
//Installed:
httpd.x86_64 0:2.4.6-90.el7.centos
...//Omit parts
[root@localhost ~]# mount.cifs //192.168.100.10/lamp-c7 /mnt / / remote mount directory
Password for root@//192.168.100.10/lamp-c7:
[root@localhost ~]# cd /mnt / / / enter the mount directory
[root@localhost mnt]# ls / / check whether the mount is successful
apr-1.6.2.tar.gz cronolog-1.6.2-14.el7.x86_64.rpm httpd-2.4.29.tar.bz2 mysql-5.6.26.tar.gz
apr-util-1.6.0.tar.gz Discuz_X2.5_SC_UTF8.zip LAMP-php5.6.txt nginx-1.12.0.tar.gz
awstats-7.6.tar.gz error.png miao.jpg php-5.6.11.tar.bz2
[root@localhost mnt]# RPM - IVH cronolog-1.6.2-14.el7.x86_.rpm / / install the software package
//Warning: cronolog-1.6.2-14.el7.x86 μ 64.rpm: head V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
//In the preparation, we should be more serious in the party and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people and the people[100%]
//Upgrading / installing
1:cronolog-1.6.2-14.el7 ################################# [100%]
[root@localhost mnt]# cd /usr/sbin / / / enter the directory
[root@localhost sbin]# ls cronolog * / / check whether the cronolog command is generated
cronolog //Successful installation
[root@localhost sbin]# vim /etc/httpd/conf/httpd.conf / / enter the edit configuration file
...//Omit parts
# prevent Apache from glomming onto all bound IP addresses.
#
Listen 192.168.144.133:80 //Set listening address
#Listen 80
#
# Dynamic Shared Object (DSO) Support
...//Omit parts
# If your host doesn't have a registered DNS name, enter its IP address here.
#
ServerName www.kgc.com:80 //Set domain name
#
# Deny access to the entirety of your server's filesystem. You must
...//Omit parts
# container, that host's errors will be logged there and not here.
#
ErrorLog "| /usr/sbin/cronolog logs/www.kgc.com.error_%Y%m%dlog"
//Call cronolog command to create error log file
#
# LogLevel: Control the number of messages logged to the error_log.
...//Omit parts
# (Combined Logfile Format) you can use the following directive.
#
CustomLog "| /usr/sbin/cronolog logs/www.kgc.com.access_%Y%m%dlog" combined
//Call the cronolog command to create an access log file
</IfModule>
...//Omit parts
:wq
[root@localhost sbin]# systemctl stop firewalld.service / / turn off the firewall
[root@localhost sbin]# setenforce 0 / / turn off enhanced security
[root@localhost sbin]# systemctl start httpd.service / / start http service
[root@localhost sbin]# cd /var/log/httpd / / / enter the log file directory
[root@localhost httpd]# ls / / view
www.kgc.com.error_20191105log //Generate log split file
[root@localhost httpd]# date / / view the current time
2019 Tuesday, November 5, 2010 20:11:57 CST
[root@localhost httpd]# date -s 11/06/19 / / set the time to the next day
2019 Wednesday, November 6, 2006:00:00 CST
[root@localhost httpd]# systemctl stop httpd.service / / stop http service
[root@localhost httpd]# systemctl start httpd.service / / start http service
[root@localhost httpd]# cd ~ / / home directory
[root@localhost ~]# cd /var/log/httpd / / / reenter the log directory
[root@localhost httpd]# ls / / view
www.kgc.com.error_20191105log www.kgc.com.error_20191106log //Generate a new log fileIntroduction of AWStats log analysis system
AWStats log analysis system
- An open source log analysis system developed by Perl
- It can be used to analyze the access logs of Apache, Samba, Vsftpd, IIS and other servers
- Information combined with crond and other scheduled task services can regularly analyze the log content
Deploy AWStats log analysis system
- Restart a Linux virtual machine, install http and DNS services, and perform general configuration
[root@localhost ~]# yum install bind httpd -y
//Loaded plug-ins: faststmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.163.com
...//Omit parts
//Installed:
bind.x86_64 32:9.11.4-9.P2.el7 httpd.x86_64 0:2.4.6-90.el7.centos
...//Omit parts
[root@localhost ~]# vim /etc/named.conf
...//Omit parts
options {
listen-on port 53 { any; };
listen-on-v6 port 53 { ::1; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
recursing-file "/var/named/data/named.recursing";
secroots-file "/var/named/data/named.secroots";
allow-query { any; };
...//Omit parts
[root@localhost ~]# vim /etc/named.rfc1912.zones
...//Omit parts
zone "kgc.com" IN {
type master;
file "kgc.com.zone";
allow-update { none; };
};
...//Omit parts
[root@localhost named]# cp -p named.localhost kgc.com.zone
[root@localhost named]# vim kgc.com.zone
$TTL 1D
@ IN SOA @ rname.invalid. (
0 ; serial
1D ; refresh
1H ; retry
1W ; expire
3H ) ; minimum
NS @
A 127.0.0.1
www IN A 192.168.144.133
:wq
[root@localhost named]# vim /etc/httpd/conf/httpd.conf
...//Omit parts
# prevent Apache from glomming onto all bound IP addresses.
#
Listen 192.168.144.133:80
#Listen 80
#
# Dynamic Shared Object (DSO) Support
...//Omit parts...
# If your host doesn't have a registered DNS name, enter its IP address here.
#
ServerName www.kgc.com:80
#
# Deny access to the entirety of your server's filesystem. You must
...//Omit parts
:wq
[root@localhost named]# systemctl stop firewalld.service
[root@localhost named]# setenforce 0
[root@localhost named]# systemctl start named
[root@localhost named]# systemctl start httpd.service - Test whether the website can be accessed in the client
- Install AWStats package
[root@localhost named]# cd /etc/httpd/ [root@localhost httpd]# mount.cifs //192.168.100.10/lamp-c7 /mnt / / mount the software storage directory in the mnt directory Password for root@//192.168.100.10/lamp-c7: / / enter directly [root@localhost httpd]# cd /mnt / / / view the hung directory [root@localhost mnt]# ls apr-1.6.2.tar.gz cronolog-1.6.2-14.el7.x86_64.rpm httpd-2.4.29.tar.bz2 mysql-5.6.26.tar.gz apr-util-1.6.0.tar.gz Discuz_X2.5_SC_UTF8.zip LAMP-php5.6.txt nginx-1.12.0.tar.gz awstats-7.6.tar.gz error.png miao.jpg php-5.6.11.tar.bz2 [root@localhost mnt]# tar zxvf awstats-7.6.tar.gz -C /opt / / / unzip the awstats tool in the opt directory awstats-7.6/ awstats-7.6/tools/ awstats-7.6/tools/awstats_buildstaticpages.pl ...//Omit parts [root@localhost mnt]# cd /opt / / / enter the opt directory [root@localhost opt]# ls awstats-7.6 rh [root@localhost opt]# mv awstats-7.6/ /usr/local/awstats / / move the extracted file to the directory / usr/local / [root@localhost opt]# cd /usr/local / / / enter the local directory [root@localhost local]# ls awstats bin etc games include lib lib64 libexec sbin share src [root@localhost local]# cd awstats / / / enter the awstats directory [root@localhost awstats]# ls docs README.md tools wwwroot [root@localhost awstats]# cd tools / / enter the tool storage directory [root@localhost tools]# ls / / view executable tools awstats_buildstaticpages.pl awstats_updateall.pl httpd_conf nginx xslt awstats_configure.pl dolibarr logresolvemerge.pl urlaliasbuilder.pl awstats_exportlib.pl geoip_generator.pl maillogconvert.pl webmin
- Create a profile for the site to be counted
[root@localhost tools]# . / awstats? Configure.pl / / execute the configuration tool
----- AWStats awstats_configure 1.0 (build 20140126) (c) Laurent Destailleur -----
This tool will help you to configure AWStats to analyze statistics for
one web server. You can try to use it to let it do all that is possible
...//Omit parts
Enter full config file path of your Web server.
Example: /etc/httpd/httpd.conf
Example: /usr/local/apache2/conf/httpd.conf
Example: c:\Program files\apache group\apache\conf\httpd.conf
Config file path ('none' to skip web server setup):
> /etc/httpd/conf/httpd.conf //Enter http profile path
...//Omit parts
-----> Need to create a new config file ?
Do you want me to build a new AWStats config/profile
file (required if first install) [y/N] ? y //Enter y to confirm
-----> Define config file name to create
What is the name of your web site or profile analysis ?
Example: www.mysite.com
Example: demo
Your web site, virtual server or profile name:
> www.kgc.com //Enter web domain name
-----> Define config file path
In which directory do you plan to store your config file(s) ?
Default: /etc/awstats
Directory path to store config file(s) (Enter for default):
> //Enter directly. The default path is to store the configuration file
...//Omit parts
Or if you have several config files and prefer having only one command:
/usr/local/awstats/tools/awstats_updateall.pl now
Press ENTER to continue...
A SIMPLE config file has been created: /etc/awstats/awstats.www.kgc.com.conf
You should have a look inside to check and change manually main parameters.
You can then manually update your statistics for 'www.kgc.com' with command:
> perl awstats.pl -update -config=www.kgc.com
You can also read your statistics for 'www.kgc.com' with URL:
> http://localhost/awstats/awstats.pl?config=www.kgc.com / / the analysis website is provided
Press ENTER to finish...
[root@localhost tools]# cd /etc/httpd/conf / / enter the http configuration file directory
[root@localhost conf]# vim httpd.conf / / edit the configuration file
...//Omit parts
# This is to permit URL access to scripts/files in AWStats directory.
#
<Directory "/usr/local/awstats/wwwroot"> //Auto generate profile
Options None
AllowOverride None
# Order allow,deny / / comment out the access permission settings
# Allow from all
Require all granted //Add items that are accessible to all
</Directory>
:wq- Modify site statistics profile
[root@localhost etc]# cd /etc/awstats / / / enter the directory of configuration awstats configuration file [root@localhost awstats]# ls / / view awstats.www.kgc.com.conf //awstats profile [root@localhost awstats]# ls /var/log/httpd / / / view the http service log file name access_log error_log [root@localhost awstats]# vim awstats.www.kgc.com.conf / / enter the awstats configuration file ...//Omit parts # Example: "/pathtotools/logresolvemerge.pl *.log |" # LogFile="/var/log/httpd/access_log" //Find the secondary entry and change the name of the log file to be analyzed # Enter the log file type you want to analyze. ...//Omit parts # Example: "C:/awstats_data_dir" # Default: "." (means same directory as awstats.pl) # DirData="/var/lib/awstats" //Find this entry and view the storage location of the analyzed data # Relative or absolute web URL of your awstats cgi-bin directory. ...//Omit parts :wq [root@localhost awstats]# cd /var/lib / / / enter the Lib directory [root@localhost lib]# mkdir awstats / / create awstats analysis data storage directory [root@localhost lib]# ls / / check whether the directory is created successfully AccountsService chrony flatpak initramfs misc PackageKit rpm-state systemd yum alsa color fprint ipa-client mlocate plymouth rsyslog tpm alternatives colord games iscsi net-snmp polkit-1 samba tuned authconfig dav gdm libvirt NetworkManager postfix selinux udisks2 awstats dbus geoclue lldpad nfs pulse setroubleshoot upower bluetooth dhclient gssproxy logrotate ntp rpcbind sss vmware certmonger dnsmasq hyperv machines os-prober rpm stateless xkb [root@localhost lib]# systemctl restart httpd.service / / restart the http service
-
Testing the awstats tool on the client
- At this time, the visited awstats did not do statistical analysis. We need to execute the update data tool to perform statistics
[root@localhost lib]# cd /usr/local/awstats / / enter the awstats tool directory [root@localhost awstats]# ls docs README.md tools wwwroot [root@localhost awstats]# cd tools / / enter the tool directory [root@localhost tools]# ls awstats_buildstaticpages.pl awstats_updateall.pl httpd_conf nginx xslt awstats_configure.pl dolibarr logresolvemerge.pl urlaliasbuilder.pl awstats_exportlib.pl geoip_generator.pl maillogconvert.pl webmin [root@localhost tools]# . / awstats [updateall. PL now / / execute the tool to update. Be careful not to add now after that Running '"/usr/local/awstats/wwwroot/cgi-bin/awstats.pl" -update -config=www.kgc.com -configdir="/etc/awstats"' to update config www.kgc.com Create/Update database for config "/etc/awstats/awstats.www.kgc.com.conf" by AWStats version 7.6 (build 20161204) From data in log file "/var/log/httpd/access_log"... Phase 1 : First bypass old records, searching new record... Searching new records from beginning of log file... Phase 2 : Now process new records (Flush history on disk after 20000 hosts)... Jumped lines in file: 0 Parsed lines in file: 48 Found 0 dropped records, Found 0 comments, Found 0 blank records, Found 0 corrupted records, //Successful update Found 0 old records, Found 48 new qualified records.
- Revisit awstats on the client
- Set up periodic scheduled tasks, automatically count access information, and optimize access paths
[root@localhost tools]# crontab -e / / set recurring scheduled tasks */5 * * * * /usr/local/awstats/tools/./awstats_updateall.pl now //Set to execute every 5 minutes ~ ~ :wq [root@localhost tools]# systemctl start crond / / start the periodic scheduled task [root@localhost tools]# cd /var/www/html / / / enter http site directory [root@localhost html]# vim aws.html / / edit web page <html> <head> <meta http-equiv=refresh content="0;url=http://Www.kgc. COM / awstats / awstats. PL? Config = www.kgc. Com "> </head> <body></body> </html> :wq //Save exit
- Test whether the optimized access path can be used in the client
