Alberta Linux 44th Lecture Apache Log Management (Optimization)

Keywords: Apache vim curl Javascript

11.22 Access log does not record static files

Edit the virtual host configuration file "httpd-vhosts.conf":

[root@adailinux ~]# vim /usr/local/apache2.4/conf/extra/httpd-vhosts.conf
......
<VirtualHost *:80>
    DocumentRoot "/data/wwwroot/111.com"
    ServerName 111.com
    ServerAlias www.example.com
    <IfModule mod_rewrite.c>
        RewriteEngine on
        RewriteCond %{HTTP_HOST} !^111.com$
        RewriteRule ^/(.*)$ http://111.com/$1 [R=301,L]
    </IfModule>
    ErrorLog "logs/111.com-error_log"
    SetEnvIf Request_URI ".*\.gif$" img
    SetEnvIf Request_URI ".*\.jpg$" img 
    SetEnvIf Request_URI ".*\.png$" img
    SetEnvIf Request_URI ".*\.bmp$" img
    SetEnvIf Request_URI ".*\.swf$" img
    SetEnvIf Request_URI ".*\.js$" img
    SetEnvIf Request_URI ".*\.css$" img  
    #Define the variable above: Define all requests for pictures as variable img
    CustomLog "logs/111.com-access_log" combined env=!img
    #"env=!img" denotes a non-img variable.The meaning of this line of commands is that the request log for the variable img is not recorded.
</VirtualHost>

Description: Define all requests to access pictures as a variable img and exclude it from the access record (log).Access records will not be generated when curl is used to access file contents in the format specified in the IMG variable after reloading.(

Extensions:

apache Logging Client Requested Domain Name

Normally, there is no need to record this item at all. After all, most of us set up access logs based on virtual hosts, but there are other situations, such as ServerName *.abc.com is such a generically parsed form that it is necessary to record which domain name the user is requesting. There is just one value in apache's LogFormat that meets this need.That is,%v is capitalized here and lowercase v records the ServerName we set up in the virtual host. This is really unnecessary.

apache logs only the specified URI

Requirements: Log requests like www.aaa.com/aaa/...(

Method: Add in httpd.conf or the associated virtual host configuration file:

SetEnvIf Request_URI "^/aaa/.*" aaa-request  
CustomLog "|/usr/local/apache/bin/rotatelogs -l /usr/local/apache/logs/aaa-access_%Y%m%d.log 86400" combined env=aaa-request

Principle and no log of static access such as pictures( http://www.lishiming.net/thread-561-1-1.html ) The same.(

Record proxy IP and real client IP in apache log

By default, the log format is:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Where%h is the IP that records visitors, if there is a layer of proxy at the front of the web, then this%h is actually the IP of the proxy machine, which is not what we want.What we need is to record the real IP of the client,'%{X-FORWARDED-FOR}i'is a field that records the real IP of the client, so the format of the log log should be changed to:

LogFormat "%h %{X-FORWARDED-FOR}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

11.23 Access log cutting

To configure

Edit the virtual host configuration file:

[root@adailinux ~]# vim /usr/local/apache2.4/conf/extra/httpd-vhosts.conf
......
<VirtualHost *:80>
    DocumentRoot "/data/wwwroot/111.com"
    ServerName 111.com
    ServerAlias www.example.com
    <IfModule mod_rewrite.c>
        RewriteEngine on
        RewriteCond %{HTTP_HOST} !^111.com$
        RewriteRule ^/(.*)$ http://111.com/$1 [R=301,L]
    </IfModule>
    ErrorLog "logs/111.com-error_log"
    SetEnvIf Request_URI ".*\.gif$" img
    SetEnvIf Request_URI ".*\.jpg$" img 
    SetEnvIf Request_URI ".*\.png$" img
    SetEnvIf Request_URI ".*\.bmp$" img
    SetEnvIf Request_URI ".*\.swf$" img
    SetEnvIf Request_URI ".*\.js$" img
    SetEnvIf Request_URI ".*\.css$" img  
    CustomLog "|/usr/local/apache2.4/bin/rotatelogs -l logs/111.com-access_%Y%m%d.log 86400" combined env=!img
    #Using the rotatelogs tool, the log is cut once a day based on system time, and the format of the log name is "111.com-access_%Y%m%d.log".
    
    #Log is cut hourly. Syntax:
    #CustomLog "|/usr/local/apache2.4/bin/rotatelogs logs/access_%Y%m%d%H.log 3600" combined

    

Description: rotatelogs is Apache's tool for cutting logs; -l denotes the use of system time (CST=Chinese time), with the -l option added, the default (UTC) time will prevail; the log file name "%Y%m%d" denotes the year and month day; and the time interval is set to one day (1day=24h=1440min=86400s).(

Detection

  • Overload profile:
[root@adailinux 111.com]# /usr/local/apache2.4/bin/apachectl -t
Syntax OK
[root@adailinux 111.com]# /usr/local/apache2.4/bin/apachectl graceful
  • Access, Detect Logs:
[root@adailinux ~]# curl -x192.168.8.131:80 111.com/http.jpg -I

[root@adailinux 111.com]# ls /usr/local/apache2.4/logs/
111.com-access_20170801.log  111.com-error_log   abc.com-error_log  error_log
111.com-access_log           abc.com-access_log  access_log         httpd.pid

Description: Use with scheduled tasks to clean up log files regularly to avoid disk space shortage due to log file accumulation.

Extensions:

rotatelogs command

Syntax: rotatelogs [ -l ] logfile [ rotationtime [ offset ]]    rotatelogs [ filesizeM ]  logfile  

Options: -l: Use local time instead of GMT time as the time base.Note: Using -l in an environment that changes GMT offsets, such as daylight saving time, can lead to unexpected results.So be sure to add -l or the log time that appears will differ by 8 hours from the actual time.( rotationtime: The interval in seconds between log file scrolls.( offset: The number of minutes of time difference from UTC.If omitted, then "0" is assumed and UTC time is used.For example, to specify the local time for a region with a UTC time difference of'-5 hours', this parameter should be'-300'.( FilesizeM: Specifies to scroll by filesizeM file size, not by time or time difference.(

Log cutting method 2:

Use the cronolog command:

One: Create a new log every day
CustomLog "|bin/cronolog logs/access_%Y%m%d.log" combined

2: Create a new log every hour
CustomLog "|bin/cronolog logs/access_%Y%m%d%h.log" combined

11.24 Static element expiration time

When the browser visits the website, it caches the static files (such as picture files, css, js files, etc.) to the local computer so that it doesn't need to download them remotely the next time it visits the website. You can customize the time to clear this part of the cache, that is, set the static element expiration time.This setting can be used to optimize your website, especially your intranet.(

Configuration

Edit Virtual Host Profile:

[root@adailinux ~]# vim /usr/local/apache2.4/conf/extra/httpd-vhosts.conf
......
<VirtualHost *:80>
    DocumentRoot "/data/wwwroot/111.com"
    ServerName 111.com
    ServerAlias www.example.com
    <IfModule mod_rewrite.c>
        RewriteEngine on
        RewriteCond %{HTTP_HOST} !^111.com$
        RewriteRule ^/(.*)$ http://111.com/$1 [R=301,L]
    </IfModule>

<IfModule mod_expires.c>
    ExpiresActive on  
    #Switch on this function
    ExpiresByType image/gif  "access plus 1 days"
    ExpiresByType image/jpeg "access plus 24 hours"
    ExpiresByType image/png "access plus 24 hours"
    ExpiresByType text/css "now plus 2 hour"
    ExpiresByType application/x-javascript "now plus 2 hours"
    ExpiresByType application/javascript "now plus 2 hours"
    ExpiresByType application/x-shockwave-flash "now plus 2 hours"
    ExpiresDefault "now plus 0 min"
    #These are the times to define different types of file caches
</IfModule>
    ErrorLog "logs/111.com-error_log"
    SetEnvIf Request_URI ".*\.gif$" img
    SetEnvIf Request_URI ".*\.jpg$" img 
    SetEnvIf Request_URI ".*\.png$" img
    SetEnvIf Request_URI ".*\.bmp$" img
    SetEnvIf Request_URI ".*\.swf$" img
    SetEnvIf Request_URI ".*\.js$" img
    SetEnvIf Request_URI ".*\.css$" img  
    CustomLog "|usr/local/apache2.4/bin/rotatelogs -l logs/111.com-access_%Y%m%d.log 86400" combined env=!img
    #Using the rotatelogs tool, the log is cut once a day based on system time, and the format of the log name is "111.com-access_%Y%m%d.log".

Description: Add mod_expires.c module content to the configuration file, leaving the rest unchanged.(

Detection

Detect grammar errors:
[root@adailinux 111.com]# /usr/local/apache2.4/bin/apachectl -t
Syntax OK

Detect if Apache configuration file opens expire module:
[root@adailinux 111.com]# /usr/local/apache2.4/bin/apachectl -M |grep expires

Note: There is no expires module detected here, so you need to edit the Apache configuration file to load the expires module.*

Configure Apache, load expires module

Edit Apache profile:

[root@adailinux 111.com]# vim /usr/local/apache2.4/conf/httpd.conf

LoadModule expires_module modules/mod_expires.so

//Load profile:
[root@adailinux 111.com]# /usr/local/apache2.4/bin/apachectl -t
Syntax OK
[root@adailinux 111.com]# /usr/local/apache2.4/bin/apachectl graceful

//Check if the module is turned on:
[root@adailinux 111.com]# /usr/local/apache2.4/bin/apachectl -M |grep expires
 expires_module (shared)

Description: Enter'/'to find the expires module and open the command line to load the module (remove #).(

Re-detect

[root@adailinux 111.com]# curl -x192.168.8.131:80 111.com/baidu.png -I
HTTP/1.1 200 OK
Date: Tue, 01 Aug 2017 10:17:36 GMT
Server: Apache/2.4.27 (Unix) PHP/5.6.30
Last-Modified: Tue, 01 Aug 2017 10:13:45 GMT
ETag: "e7a-555ae670b0840"
Accept-Ranges: bytes
Content-Length: 3706
Cache-Control: max-age=86400
Expires: Wed, 02 Aug 2017 10:17:36 GMT
Content-Type: image/png

Description: Current time "Date: Tue, 01 Aug 2017 10:17:36 GMT", cache time "Cache-Control: max-age=86400s", expiration time "Expires: Wed, 02 Aug 2017 10:17:36 GMT", that is, png format picture cache time is 1 day.

Posted by ikebaldo on Sat, 08 Jun 2019 10:16:03 -0700