ELK schema log processing logic:

1. When Filebeat is installed in the business layer, it will automatically obtain the domain name and environment information of the host running site, add channel and env tags, and take the value of channel as the topic information of kafka

2. Kafka receives the new field and Topic information of Filebeat, and automatically creates Topic information to wait for logstash consumption

3. Logstash automatically generates input and output configuration according to script

The topic here must be consistent with the channel of filebeat.

Demonstration:

filebeat layer:

- type: log
  processors:
  - add_fields:
      fields:
        env: "prod"         ## ansible calls Python to automatically judge and generate based on network segment information
        ip: "10.12.11.27"   ## ansible calls Python to automatically judge and generate based on network segment information
        apptype: "service"  ## ansible calls Python to automatically judge and generate according to the domain name
        channel: "cms.prod.tarscorp.com"   ##ansible calls Python to generate from site directory
  enabled: true
  paths:
    - /data1/logs/cms.prod.tarscorp.com/*.log

output.kafka:

  codec.json:
    pretty: true
    escape_html: false

  hosts: ["kafka1.mgt.tarscorp.com:9092", "kafka2.mgt.tarscorp.com:9092", "kafka3.mgt.tarscorp.com:9092"]

  topic: 'cms.prod.tarscorp.com'        ## topic and channel are from the same data
  partition.round_robin:
    reachable_only: false

  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

Kafka layer: ignore, cluster + enable automatic Topic creation

logstash layer:

vim prod-input.conf    ##Input information

kafka {
      topics => "cms.prod.tarscorp.com"     ## topic information in kafka
      bootstrap_servers => "kafka1.mgt.tarscorp.com:9092,kafka2.mgt.tarscorp.com:9092,kafka3.mgt.tarscorp.com:9092"
      decorate_events => false
      group_id => "logstash-tars"
      # consumer_threads => 5
      client_id => "mgt-elk-logstash1-prod"
      codec => "json"
      add_field => {"topic"=>"cms.prod.tarscorp.com"}   ##This is mainly for the convenience of logstash
   }

vim prod-javasite.conf    ##Output information
if [topic] == "cms.prod.tarscorp.com" {
                elasticsearch {
                        hosts => ["mgt-elk-esmaster1:9200", "mgt-elk-esmaster2:9200", "mgt-elk-esmaster3:9200"]
                        manage_template => false
                        index => "prod-javasite-%{+YYYY.MM.dd}"
                }
        }

Note: the technology stack here is Java spring, so all Java sites will be placed under the index of [prod javasite -% {+ yyyy. Mm. DD}], Because there are many configuration links, when the site is put on the shelf, the distribution machine is used to deploy the service first, and then ansible deploys the filebeat. Ansible will obtain the server network segment and site information through Python script, supplement the channel, Topic, apptype, env and ip tag generation configuration through templates, realize automatic judgment and reduce the burden of operation and maintenance participation.

Use script generation for example:

./add_info.py --env prod --topic cms.prod.tarscorp.com --module javasite

vim add_info.py

#!/usr/bin/env python3

import os,sys,argparse

parser = argparse.ArgumentParser(description='Logstash configuration file add tools')

parser.add_argument('--env',type=str,required=True,help='environmental information ')
parser.add_argument('--topic',type=str,required=True,help='Topic information')
parser.add_argument('--module',type=str,required=True,help='Module information')

args = parser.parse_args()

env_info = args.env
topic_name = args.topic
module_info = args.module
date = "%{+YYYY.MM.dd}"

template_input = '''
   kafka {
      topics => "%s"
      bootstrap_servers => "kafka1.mgt.tarscorp.com:9092,kafka2.mgt.tarscorp.com:9092,kafka3.mgt.tarscorp.com:9092"
      decorate_events => false
      group_id => "logstash-tars"
      # consumer_threads => 5
      client_id => "mgt-elk-logstash1-dev"
      codec => "json"
      add_field => {"topic"=>"%s"}
   }
}
''' %(topic_name,topic_name)

template_output = '''
    if [topic] == "%s" {
        elasticsearch {
            hosts => ["mgt-elk-esmaster1:9200", "mgt-elk-esmaster2:9200", "mgt-elk-esmaster3:9200"]
            manage_template => false
            index => "%s-%s-%s"
        }
    }
}
''' %(topic_name,env_info,module_info,date)

init_input = '''
input {

'''
init_output = '''
output {

'''

path_home = "/etc/logstash/conf.d/"
input_file = "/etc/logstash/conf.d/%s-input.conf" % (env_info)
output_file = "/etc/logstash/conf.d/%s-%s.conf" % (env_info, module_info)

if os.path.exists(path_home) == False:
    print('Please be there. logstash The host runs the script')
    exit(code=255)

if os.path.exists(input_file) == False:
    with open(input_file, mode='w', encoding='utf-8') as f:
        f.write(init_input)

if os.path.exists(output_file) == False:
    with open(output_file, mode='w', encoding='utf-8') as f:
        f.write(init_output)

with open(input_file,mode='rb+') as f:
    f.seek(-2,2)
    f.write(template_input.encode('utf-8'))

with open(output_file,mode='rb+') as f:
    f.seek(-2,2)
    f.write(template_output.encode('utf-8'))