What is ELK?
ELK is a complete set of log collection and front-end display solution provided by elastic company. It is the acronym of three products, namely ElasticSearch, Logstash and Kibana.
Among them, Logstash is responsible for processing logs, such as filtering logs, formatting logs, etc.; ElasticSearch has a strong text search ability, so it serves as a storage container for logs; Kibana is responsible for the display of the front end.
The ELK architecture is as follows:
filebeat is added to collect logs from different clients and then pass them to Logstash for unified processing.
Construction of ELK
Because ELK is three products, you can choose to install them one by one.
Here you choose to use Docker to install ELk.
Docker can also choose to download the images of these three products and run them separately when installing elk, but this time, you can use the three in one image of downloading elk directly to install.
Therefore, first of all, you need to ensure that you have a Docker running environment. For the construction of Docker running environment, please refer to: https://blog.csdn.net/qq13112...
Pull mirror image
With the Docker environment, run the command on the server:
docker pull sebp/elk
This command is to download the elk three in one image from the Docker warehouse, with a total size of more than two gigabytes. If the download speed is too slow, you can replace the source address of the Docker warehouse with the domestic source address.
When the download is complete, view the image:
docker images
Logstash configuration
Create beats-input.conf in the / usr/config/logstash directory for log input:
input { beats { port => 5044 } }
Create a new output.conf for log output from Logstash to ElasticSearch:
output { elasticsearch { hosts => ["localhost"] manage_template => false index => "%{[@metadata][beat]}" } }
The index is the index after outputting to ElasticSearch.
Running container
With the image, you can start it directly:
docker run -d -p 5044:5044 -p 5601:5601 -p 9203:9200 -p 9303:9300 -v /var/data/elk:/var/lib/elasticsearch -v /usr/config/logstash:/etc/logstash/conf.d --name=elk sebp/elk
-d means to run the container in the background;
-p means host port: container port, that is, the port used in the container is mapped to a port on the host. The default port of ElasticSearch is 9200 and 9300. Since three ElasticSearch instances have been running on my machine, the mapping port is modified here.
-v means the file folder of the host: the file folder of the container. Here, mount the data of elasticsearch in the container to / var/data/elk of the host to prevent data loss after the container restarts. Also, mount the configuration file of logstash to / usr/config/logstash directory of the host.
--Name means to name the container, which is more convenient to operate later.
If you have built ElasticSearch before, you will find that there are various errors in the process of building, but there are no errors in the process of using docker to build elk.
To view containers after running:
docker ps
To view the container log:
docker logs -f elk
Enter container:
docker exec -it elk /bin/bash
Restart the container after modifying the configuration:
docker restart elk
View kinaba
Browser input http://my_host:5601/
You can see the kinaba interface. At this time, there is no data in ElasticSearch. You need to install Filebeat to collect data into elk.
Filebeat building
Filebeat is used to collect data and report it to Logstash or ElasticSearch. Download filebeat on the server where you need to collect logs and extract it for use.
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.1-linux-x86_64.tar.gz
tar -zxvf filebeat-6.2.1-linux-x86_64.tar.gz
Modify profile
Enter filebeat and modify filebeat.yml.
filebeat.prospectors: - type: log #Configuration needs to be set to true to take effect enabled: true path: #Configure the log path to be collected - /var/log/*.log #You can type a tag and use it by category. tag: ["my_tag"] #type corresponding to ElasticSearch document_type: my_type setup.kibana: #Here is the ip and port of kibana, i.e. kibana:5601 host: "" output.logstash: #Here is the ip and port of logstash, i.e. logstash:5044 host: [""] #It needs to be set to true, otherwise it will not take effect. enabled: true #If you want to collect data directly from Filebeat to ElasticSearch, you can configure the relevant configuration of output.elasticsearch.
Run Filebeat
Function:
./filebeat -e -c filebeat.yml -d "publish"
At this point, you can see that Filebeat will send the log under the configured path to Logstash; then in elk, Logstash will send the log to ElasticSearch after processing the data. But what we want to do is analyze the data through elk, so the data imported to ElasticSearch must be in JSON format.
This is the format of my previous single log:
2019-10-22 10:44:03.441 INFO rmjk.interceptors.IPInterceptor Line:248 - {"clientType":"1","deCode":"0fbd93a286533d071","eaType":2,"eaid":191970823383420928,"ip":"xx.xx.xx.xx","model":"HONOR STF-AL10","osType":"9","path":"/applicationEnter","result":5,"session":"ef0a5c4bca424194b29e2ff31632ee5c","timestamp":1571712242326,"uid":"130605789659402240","v":"2.2.4"}
It's not easy to analyze after importing. Later, I thought of using grok in Logstash's filter to process the log and make it into JSON format before importing it into ElasticSearch. However, because the parameters in my log are not fixed, it's too difficult to find out, so I used Logback instead. I formatted the log directly into JSON and sent it by Filebeat.
Logback configuration
My project is Spring Boot. Add dependency in the project:
<dependency> <groupId>net.logstash.logback</groupId> <artifactId>logstash-logback-encoder</artifactId> <version>5.2</version> </dependency>
Then add logback.xml in the resource directory of the project:
<?xml version="1.0" encoding="UTF-8"?> <configuration> <!-- Explain: 1,Log level and file The log record adopts hierarchical record, the level corresponds to the log file name, and the log information of different levels is recorded in different log files. For example: error Level recorded to log_error_xxx.log or log_error.log(The file is the currently logged log file), and log_error_xxx.log To archive logs, Log files are recorded by date. In the same day, if the log file size is equal to or greater than 2 M，Then press 0, 1, 2...Order named separately for example log-level-2013-12-21.0.log The same is true for other levels of logging. 2,File path If it is used for development and test, it can be Eclipse To run the project Eclipse Installation path lookup for logs Folder to relative path../logs. If deployed to Tomcat Next, then Tomcat Lower logs In file 3,Appender FILEERROR Corresponding error Level, filename with log-error-xxx.log Formal nomenclature FILEWARN Corresponding warn Level, filename with log-warn-xxx.log Formal nomenclature FILEINFO Corresponding info Level, filename with log-info-xxx.log Formal nomenclature FILEDEBUG Corresponding debug Level, filename with log-debug-xxx.log Formal nomenclature stdout Output the log information to the control for the convenience of development and test --> <contextName>service</contextName> <property name="LOG_PATH" value="logs"/> <!--Set system log directory--> <property name="APPDIR" value="doctor"/> <!-- Loggers, date scrolling --> <appender name="FILEERROR" class="ch.qos.logback.core.rolling.RollingFileAppender"> <!-- Path and filename of the log file being logged --> <file>${LOG_PATH}/${APPDIR}/log_error.log</file> <!-- Rolling strategy for loggers, by date, by size --> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <!-- The path to the archived log file, for example, today is 2013-12-21 Log, the current write log file path is file Node, you can compare this file with file Specifies that the file path is set to a different path to set the current log file or archive log file to a different directory. And 2013-12-21 The log file for is in the fileNamePattern Appoint.%d{yyyy-MM-dd}Specify the date format,%i Specified index --> <fileNamePattern>${LOG_PATH}/${APPDIR}/error/log-error-%d{yyyy-MM-dd}.%i.log</fileNamePattern> <!-- In addition to logging by log, it is configured that the log file cannot exceed 2 M，If more than 2 M，The log file will start with index 0. Name the log file, for example log-error-2013-12-21.0.log --> <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP"> <maxFileSize>2MB</maxFileSize> </timeBasedFileNamingAndTriggeringPolicy> </rollingPolicy> <!-- Log by appending method --> <append>true</append> <!-- Format of log file --> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level %logger Line:%-3L - %msg%n</pattern> <charset>utf-8</charset> </encoder> <!-- This log file records only info Grade --> <filter class="ch.qos.logback.classic.filter.LevelFilter"> <level>error</level> <onMatch>ACCEPT</onMatch> <onMismatch>DENY</onMismatch> </filter> </appender> <!-- Loggers, date scrolling --> <appender name="FILEWARN" class="ch.qos.logback.core.rolling.RollingFileAppender"> <!-- Path and filename of the log file being logged --> <file>${LOG_PATH}/${APPDIR}/log_warn.log</file> <!-- Rolling strategy for loggers, by date, by size --> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <!-- The path to the archived log file, for example, today is 2013-12-21 Log, the current write log file path is file Node, you can compare this file with file Specifies that the file path is set to a different path to set the current log file or archive log file to a different directory. And 2013-12-21 The log file for is in the fileNamePattern Appoint.%d{yyyy-MM-dd}Specify the date format,%i Specified index --> <fileNamePattern>${LOG_PATH}/${APPDIR}/warn/log-warn-%d{yyyy-MM-dd}.%i.log</fileNamePattern> <!-- In addition to logging by log, it is configured that the log file cannot exceed 2 M，If more than 2 M，The log file will start with index 0. Name the log file, for example log-error-2013-12-21.0.log --> <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP"> <maxFileSize>2MB</maxFileSize> </timeBasedFileNamingAndTriggeringPolicy> </rollingPolicy> <!-- Log by appending method --> <append>true</append> <!-- Format of log file --> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level %logger Line:%-3L - %msg%n</pattern> <charset>utf-8</charset> </encoder> <!-- This log file records only info Grade --> <filter class="ch.qos.logback.classic.filter.LevelFilter"> <level>warn</level> <onMatch>ACCEPT</onMatch> <onMismatch>DENY</onMismatch> </filter> </appender> <!-- Loggers, date scrolling --> <appender name="FILEINFO" class="ch.qos.logback.core.rolling.RollingFileAppender"> <!-- Path and filename of the log file being logged --> <file>${LOG_PATH}/${APPDIR}/log_info.log</file> <!-- Rolling strategy for loggers, by date, by size --> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <!-- The path to the archived log file, for example, today is 2013-12-21 Log, the current write log file path is file Node, you can compare this file with file Specifies that the file path is set to a different path to set the current log file or archive log file to a different directory. And 2013-12-21 The log file for is in the fileNamePattern Appoint.%d{yyyy-MM-dd}Specify the date format,%i Specified index --> <fileNamePattern>${LOG_PATH}/${APPDIR}/info/log-info-%d{yyyy-MM-dd}.%i.log</fileNamePattern> <!-- In addition to logging by log, it is configured that the log file cannot exceed 2 M，If more than 2 M，The log file will start with index 0. Name the log file, for example log-error-2013-12-21.0.log --> <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP"> <maxFileSize>2MB</maxFileSize> </timeBasedFileNamingAndTriggeringPolicy> </rollingPolicy> <!-- Log by appending method --> <append>true</append> <!-- Format of log file --> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level %logger Line:%-3L - %msg%n</pattern> <charset>utf-8</charset> </encoder> <!-- This log file records only info Grade --> <filter class="ch.qos.logback.classic.filter.LevelFilter"> <level>info</level> <onMatch>ACCEPT</onMatch> <onMismatch>DENY</onMismatch> </filter> </appender> <appender name="jsonLog" class="ch.qos.logback.core.rolling.RollingFileAppender"> <!-- Path and filename of the log file being logged --> <file>${LOG_PATH}/${APPDIR}/log_IPInterceptor.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <fileNamePattern>${LOG_PATH}/${APPDIR}/log_IPInterceptor.%d{yyyy-MM-dd}.log</fileNamePattern> </rollingPolicy> <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder"> <jsonFactoryDecorator class="net.logstash.logback.decorate.CharacterEscapesJsonFactoryDecorator"> <escape> <targetCharacterCode>10</targetCharacterCode> <escapeSequence>\u2028</escapeSequence> </escape> </jsonFactoryDecorator> <providers> <pattern> <pattern> { "timestamp":"%date{ISO8601}", "uid":"%mdc{uid}", "requestIp":"%mdc{ip}", "id":"%mdc{id}", "clientType":"%mdc{clientType}", "v":"%mdc{v}", "deCode":"%mdc{deCode}", "dataId":"%mdc{dataId}", "dataType":"%mdc{dataType}", "vid":"%mdc{vid}", "did":"%mdc{did}", "cid":"%mdc{cid}", "tagId":"%mdc{tagId}" } </pattern> </pattern> </providers> </encoder> </appender> <!-- Color log --> <!-- Color log dependent rendering class --> <conversionRule conversionWord="clr" converterClass="org.springframework.boot.logging.logback.ColorConverter"/> <conversionRule conversionWord="wex" converterClass="org.springframework.boot.logging.logback.WhitespaceThrowableProxyConverter"/> <conversionRule conversionWord="wEx" converterClass="org.springframework.boot.logging.logback.ExtendedWhitespaceThrowableProxyConverter"/> <!-- Color log format --> <property name="CONSOLE_LOG_PATTERN" value="${CONSOLE_LOG_PATTERN:-%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}"/> <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> <!--encoder The default configuration is PatternLayoutEncoder--> <encoder> <pattern>${CONSOLE_LOG_PATTERN}</pattern> <charset>utf-8</charset> </encoder> <!--This log appender It is used for development. Only the lowest level is configured. The log level output by the console is the log information greater than or equal to this level.--> <filter class="ch.qos.logback.classic.filter.ThresholdFilter"> <level>debug</level> </filter> </appender> <!-- Specifies the logging level of a package in the project when there is a logging action --> <!-- rmjk.dao.mappe For the root package, that is to say, the permissions for all log operations that occur under the root package are DEBUG --> <!-- The level is [from high to low]: FATAL > ERROR > WARN > INFO > DEBUG > TRACE --> <logger name="rmjk.dao.mapper" level="DEBUG"/> <logger name="rmjk.service" level="DEBUG"/> <!--Display log--> <logger name="org.springframework.jdbc.core" additivity="false" level="DEBUG"> <appender-ref ref="STDOUT"/> <appender-ref ref="FILEINFO"/> </logger> <!-- Printing json Journal --> <logger name="IPInterceptor" level="info" additivity="false"> <appender-ref ref="jsonLog"/> </logger> <!-- In the production environment, configure this level to the appropriate level to avoid too many log files or affecting program performance --> <root level="INFO"> <appender-ref ref="FILEERROR"/> <appender-ref ref="FILEWARN"/> <appender-ref ref="FILEINFO"/> <!-- The production environment will stdout,testfile Remove --> <appender-ref ref="STDOUT"/> </root> </configuration>
The key points are:
<logger name="IPInterceptor" level="info" additivity="false"> <appender-ref ref="jsonLog"/> </logger>
Introduce slf4j in the file to be printed:
private static final Logger LOG = LoggerFactory.getLogger("IPInterceptor");
Put the information to be printed in MDC:
MDC.put("ip", ipAddress); MDC.put("path", servletPath); MDC.put("uid", paramMap.get("uid") == null ? "" : paramMap.get("uid").toString());
At this time, if LOG.info("msg") is used, the printed content will be input into the message of the log, and the log format is as follows:
Modify Logstash configuration
Modify beats-input.conf in the directory / usr/config/logstash:
input { beats { port => 5044 codec => "json" } }
Only one sentence of codec = > JSON is added, but Logstash will parse the input according to the JSON format.
Because of the configuration changes, restart elk:
docker restart elk
In this way, when our logs are generated and imported into elk using Filebeat, we can analyze the logs through Kibana.
Turn to praise is the biggest encouragement