Rsyslog

Everyone in the company is following the trend of learning es's log scheme. I've been struggling from three years ago to now. Filebeat + kafka + logstash+ es, at that time, I also wanted to build another storm. Later, I found that just a little log volume and a hammer kafka were used. Many of the tools that had been used were damaged by the Internet. In general, how can small businesses use the high concurrency scheme of others? Just go back to the original heart and study these technical tools used by the original people step by step; Syslog is a very useful log collection tool, followed by rsyslog and syslog ng. Learn about rsyslog first. It's good for small businesses.

introduce

The core is the rsyslog daemons. This service is responsible for listening to the log information under Linux and appending the log information to the corresponding log file, usually under the / var/log directory. It can also pass the log information through the network protocol RFC 3195 , file, zmq3, klog, etc.) send the logs to another Linux server, or store the logs in MySQL or Oracle database (supported output includes ES, mongo, MySQL), and of course, a bunch of features, such as email sending, compression, etc,. At present, the large version of rsyslog is v8, and the default installation for the computer is v7. The main processing process is the same as other log processing schemes, input - > Modify - > output. There are several plug-ins available in each link.

To configure

In general, the rsyslog configuration files are located in the / etc/rsyslog.conf and / etc/rsyslog.d / folders

Global configuration

Load some common modules

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal

Set default log format

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

rule Rules are composed of selectors and actions. Selectors indicate the source and log level, while actions indicate how to handle the corresponding logs. Here are some examples

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog

Horizontal line - description is asynchronous write

custom

Queues in Rsyslog

The following figure shows the rsyslog queue. Refer to Official introduction

As can be seen from the figure, the data will go through two queues, one is the main queue and the other is the action queue

To configure

Modular

Module (load="imudp")

input

Input (type="imudp" port="514")

filter

If $fromhost-ip == "172.19.1.135" then {
Action (type="omfile" File="/var/log/network1.log")
}

Rule set

Input (type="imudp" port="514" ruleset="rs1")

control logic

if ($msg contains "word" and $source == "123.123.123.123") then {
    action()
    action()
} else {
    action()
    action()
}

Reference resources

guide

https://www.rsyslog.com/category/guides-for-rsyslog/

pri

The first part of the syslog protocol is a number referenced by angle brackets, for example: < 182 > The size of the number ranges from 0 to 255, which is a number expressed in 1 byte, and includes two parts: The lower three bits: (0 ~ 7) are called Severity

        Numerical         Severity
          Code
 
           0       Emergency: system is unusable
           1       Alert: action must be taken immediately
           2       Critical: critical conditions
           3       Error: error conditions
           4       Warning: warning conditions
           5       Notice: normal but significant condition
           6       Informational: informational messages
           7       Debug: debug-level messages

High 5 bits (after 3 right shifts): (0 ~ 31) is called: Facility

       Numerical             Facility
          Code
 
           0             kernel messages
           1             user-level messages
           2             mail system
           3             system daemons
           4             security/authorization messages (note 1)
           5             messages generated internally by syslogd
           6             line printer subsystem
           7             network news subsystem
           8             UUCP subsystem
           9             clock daemon (note 2)
          10             security/authorization messages (note 1)
          11             FTP daemon
          12             NTP subsystem
          13             log audit (note 1)
          14             log alert (note 1)
          15             clock daemon (note 2)
          16             local use 0  (local0)
          17             local use 1  (local1)
          18             local use 2  (local2)
          19             local use 3  (local3)
          20             local use 4  (local4)
          21             local use 5  (local5)
          22             local use 6  (local6)
          23             local use 7  (local7)