Previous articles EFK Follow-up of Logging System: fluent-bit Service Independence It completes fluent-bit acquisition, fluentd forwarding to kafka, and then to elastic search. Later, it is mentioned that the server logs should be synchronized to the fluent-bit environment. This can be accomplished by incremental synchronization through rsync, without recording in detail. Now we mainly record the alarm log monitoring and sending messages in kafka. The process of notification.

fluentd configuration filters error level logs

<filter fb.dapeng>
  @type record_transformer
  enable_ruby
  <record>
    tag ${record["fbKey"].split('/')[3]}
  </record>
  remove_keys fbKey
</filter>

<match fb.dapeng>
  @type copy
  <store ignore_error>
    @type rewrite_tag_filter
    <rule>
      key     level
      pattern /^ERROR$/
      tag     error.fb.dapeng
    </rule>
  </store>
  <store ignore_error>
    @type kafka_buffered
    brokers kafka The server ip:9092
    topic_key efk
    buffer_type file
    buffer_path /tmp/buffer
    flush_interval 5s
    default_topic efk
    output_data_type json
    compression_codec gzip
    max_send_retries 3
    required_acks -1
    discard_kafka_delivery_failed true
  </store>
</match>

<match error.fb.dapeng>
    @type kafka_buffered
    brokers kafka The server ip:9092
    topic_key efk_error
    buffer_type file
    buffer_path /tmp/buffer_error
    flush_interval 5s
    default_topic efk_error
    output_data_type json
    compression_codec gzip
    max_send_retries 3
    required_acks -1
    discard_kafka_delivery_failed true
</match>

• copy: copy each event to multiple outputs, store is equivalent to match
• rewrite_tag_filter: Rewrite tag according to rule rule rule for matching event, send message with new tag, and re-process from top to bottom, so pay attention to rewrite tag does not match the current match, otherwise it will fall into a dead cycle.

Here matches the message tag fb.dapeng:

1. Messages matching level as ERROR rewrite tag as error.fb.dapeng,
2. Messages are sent directly to the topic of kafka: efk

For the message rewritten as error.fb.dapeng in tag 1, send the message to the topic of kafka: efk_error

In this way, elastic search only consumes kafka's topic: efk, which contains all levels of log information.
For alarm monitoring monitor, only the topic of consuming kafka: efk_error, which is ERROR-level log, is used.

Note: The rewrite_tag_filter plug-in needs to be installed and the Dockerfile of fluentd needs to be modified to rebuild the image

FROM fluent/fluentd:v1.2
#Add es plug-in, kafka plug-in, rewrite_tag_filter plug-in
RUN  fluent-gem install fluent-plugin-elasticsearch
RUN  fluent-gem install fluent-plugin-kafka
RUN  fluent-gem install fluent-plugin-rewrite-tag-filter
CMD exec fluentd -c /fluentd/etc/${FLUENTD_CONF} -p /fluentd/plugins $FLUENTD_OPT

monitor project consumes kafka messages

The logic of kafka consumption processing is as follows:

1. First, parse the message json and take out session Tid.
2. For the same session Tid, only the first error message is recorded (session Tid caches for 10s, then clears, usually the same call error will not interval for 10s)
3. Nail Group Message Notification

@Component
@Slf4j
public class EfkAlarmConsumer {

  @Resource
  EfkAlarmApp efkAlarmApp;

  private final Map<String, Long> cache = new ConcurrentHashMap<>();

  private final Timer timer = new Timer();

  @KafkaListener(topics = {"efk_error"}, groupId = "efk_error_consumer")
  public void processEfkAlarm(ConsumerRecord<String, String> record) {
    String json = record.value();

    Log l = resolveLog(json);
    if (null == l) {
      log.error("illegitimate message: {}", json);
    } else {
      log.debug("receive messages Log: {}", l);
      processLog(l);
    }
  }

  private void processLog(Log l) {
    final String tid = l.getSessionTid();
    Long t = cache.get(tid);
    if (t == null) {
      cache.put(tid, System.currentTimeMillis());

      // Data Clearance of tid after 10s
      timer.schedule(new TimerTask() {
        @Override
        public void run() {
          cache.remove(tid);
        }
      }, 10000);

      String currIndex = String.format("dapeng_log_index-%s", new SimpleDateFormat("yyyy.MM.dd").format(new Date()));
      // Hair nails...
      String text = String.format("%s", l.getMessage());
      String title = String.format("[%s] %s: %s[%s] ", efkAlarmApp.getDingTag(), l.getLogtime(), l.getTag(), l.getHostname());
      String url = String.format(AppConst.ESHEAD_LINK_URI, currIndex, l.getSessionTid());
      
      DingService.send(efkAlarmApp.getDingWebHook(), msg(text, title, url));
    }
  }

  private Log resolveLog(String json) {
    Log l = null;
    try {
       l = JSON.parseObject(json, Log.class);
    } catch (Throwable e) {
      log.error("Message transformation exception", e);
    }
    return l;
  }

  private String msg(String text, String title, String url) {
    return String.format(
        "{\n" +
        "  \"msgtype\": \"link\", \n" +
        "  \"link\": {\n" +
        "    \"text\": \"%s\", \n" +
        "    \"title\": \"%s\", \n" +
        "    \"picUrl\": \"\",\n" +
        "    \"messageUrl\": \"%s\"\n" +
        "  }\n" +
        "}",
        text, title, url);
  }

}

Message Link Jump

link format ESHEAD_LINK_URI:

public class AppConst {

  public static final String ES_BASE_URI = "elasticsearch Server Access Address";
  public static final String ESHEAD_BASE_URI = "elasticsearch-head Page Access Address";
  public static final String ESHEAD_LINK_URI = ESHEAD_BASE_URI + "?curr_index=%s&sessionTid=%s&base_uri=" + ES_BASE_URI;
}

Modify index.html in the elastic search-head project

                <script>
                        window.onload = function() {
                                if(location.href.contains("/_plugin/")) {
                                        var base_uri = location.href.replace(/_plugin\/.*/, '');
                                }
                                var args = location.search.substring(1).split("&").reduce(function(r, p) {
                                        r[decodeURIComponent(p.split("=")[0])] = decodeURIComponent(p.split("=")[1]); return r;
                                }, {});
                                new app.App("body", {
                                        id: "es",
                                        base_uri: args["base_uri"] || base_uri,
                                        auth_user : args["auth_user"] || "",
                                        auth_password : args["auth_password"],
                                        dashboard: args["dashboard"]
                                });

<!-- If the parameter band sessionTid, The instructions are jumped from the nail message link, Pages do special operations -->
if ("sessionTid" in args) {
  $(".uiApp-headerNewMenuItem")[0].click();

  var t1 = setInterval(function() {
    if ($(".uiIndexSelector-select > select").length > 0) {
      clearInterval(t1);

      $(".uiIndexSelector-select > select").last()[0].value = args["curr_index"];
      $(".uiIndexSelector-select > select").last().change();

      var t2 = setInterval(function() {
        if ($(".field").length > 0) {
          clearInterval(t2);
          $(".field").last()[0].value = "dapeng_log.sessionTid";
          $(".field").last().change();

          var t3 = setInterval(function() {
            if ($(".qual").length > 0) {
              clearInterval(t3);
              $(".qual").last()[0].value =  args["sessionTid"];
              $("button").last().click();
            }
          }, 20);
        }
      }, 20);
    }

  }, 20);
}
                        };
                </script>

Modify the configuration of elastic search-head in dc-all.yml file and add the mount of index.html

- /data/workspace/elasticsearch-head/index.html:/usr/src/app/index.html

OK, restart elastic search-head

The nail message is as follows:

Click Message Jump es-head: