Log Processing Mechanism

• Let's first look at the mechanism of docker log processing. When a container is started, it's actually a sub-process of docker daemon. docker daemon can get the standard output of the process in your container. When it gets the standard output, it will process it through its own LogDriver module, the way LogDriver supports it. Many can be written to local files (default), can be sent to syslog and so on.

• The docker defaults to the standard output of the mobile application stored in a json.log file in a format similar to the following:

  {"log": "root@c835298de6dd:/# ls\r\n", "stream": "stdout", "time": "xxoo.155863426Z"}
  {"log":"bin boot dev\u0009etc home lib\u0009lib64 media mnt opt\u0009proc root run sbin selinux\u0009srv sys tmp usr var\r\n"}
  

  Store one line at a time as a json data store. docker's log storage mode is configurable. Specific parameters can be configured by log-driver when running run to start the container. Specific configuration can refer to log-driver.

• Docker uses json-file driver as log driver by default, while gelf is the log driver we need to use. When there are too many containers, or when we deploy dockers using swarm-like clusters, all kinds of logs are scattered in json.log files. When we look for problems or make relevant statistics, scattered logs are very unfriendly to us. We need a tool that can centrally manage docker logs, which is graylog.

Graylog

• docker supports graylog protocol natively and sends logs directly to graylog (through gelf protocol);
• graylog officially provides support for deploying itself in docker.

Graylog officially provides dockerfile for us to quickly deploy the log system on the docker. In this docker hub address, docker-compose.yml is also provided to rapidly deploy the entire graylog stack, including MongoDB and elastic search, without the need to deploy separately.

https://hub.docker.com/r/graylog/graylog

graylog deployment

• Create a directory to deploy graylog. This article assumes that the directory is / root/graylog. All of the following operations are done in / root/graylog.

• Initialize directories and configuration files

  # Create a data catalog
  mkdir -p ./graylog/data
  # Create a configuration file directory
  mkdir -p ./graylog/config
  cd ./graylog/config
  # Download the official recommended configuration file directly
  wget https://raw.githubusercontent.com/Graylog2/graylog-docker/2.5/config/graylog.conf
  # Log Profile
  wget https://raw.githubusercontent.com/Graylog2/graylog-docker/2.5/config/log4j2.xml
  

• Modify the root_timezone in the downloaded graylog.conf to GMT+0800 China Time Zone

  root_timezone=Etc/GMT-8
  

• Create a new docker-compose.yml for docker-compose quick start-up to complete the service. It should be noted that due to the docker-compose content, we store it in the current directory of the same level as an attachment.

• Start the whole service

  docker-compose up
  

• If there are no problems, you will see the terminal output message of graylog web server started. Visit http:/{server}: 9000 and you will see Graylog's web interface. Use the username admin and password admin to log in to the background. Deployment is completed.

Complete docker-compose file

version: '2'
services:
  mongodb:
    image: mongo:3
    volumes:
      - /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime
      - mongo_data:/data/db
  elasticsearch:
    image: elasticsearch:6.6.2
    volumes:
      - /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime
      - es_data:/usr/share/elasticsearch/data
# You need to add a jvm.options file locally and specify the garbage collector as G1GC, otherwise you cannot start successfully.
      - /Users/zhangxufeng/xufeng.zhang/docker/conf-compose/graylog/graylog/jvm.options:/usr/share/elasticsearch/config/jvm.options
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - xpack.security.enabled=false
      - xpack.watcher.enabled=false
      - xpack.monitoring.enabled=false
      - xpack.security.audit.enabled=false
      - xpack.ml.enabled=false
      - xpack.graph.enabled=false
      - ES_JAVA_OPTS=-Xms512m -Xmx512m -XX:+UseG1GC
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 512M
  graylog:
    image: graylog/graylog:2.5
    volumes:
      - /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime
      - graylog_journal:/usr/share/graylog/data/journal
      - ./graylog/config:/usr/share/graylog/data/config
    environment:
      - GRAYLOG_PASSWORD_SECRET=admin_zxfwy1314_
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      - GRAYLOG_WEB_ENDPOINT_URI=http://127.0.0.1:9000/api
    links:
      - mongodb:mongo
      - elasticsearch
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      - 9000:9000
      - 514:514
      - 514:514/udp
      - 12201:12201
      - 12201:12201/udp
volumes:
  mongo_data:
    driver: local
  es_data:
    driver: local
  graylog_journal:
    driver: local

Graylog System Configuration

• input configuration

  • Graylog's log collection is accomplished by defining the input object. In the graylog web management interface, the input object configuration is entered as follows. The GELF UDP protocol is selected to create a new input.

    

  • Fill in the relevant attributes, create new ones, save them, and you can start collecting logs:

    

• docker configuration

  • If docker is started from the command line, the following parameters can be added to the run command:

    docker run --log-driver=gelf --log-opt gelf-address=udp://graylog Server Address}: 12201 -- log-opt tag= <Current Container Service Label, Used for Classification when graylog Queries> <IMAGE> <Running Command>
    

    Examples:

    docker run -d --log-driver=gelf --log-opt gelf-address=udp://localhost:12201 --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" busybox sh -c 'while true; do echo "Hello, this is A"; sleep 10; done;'
    

  • If you use the docker-compose command, you can add configuration to docker-compose.yml. Take the NGINX container for example:

    version: '2'
    services:
      nginx:
        image: nginx:latest
        ports:
          - "80:80"
        logging:
          driver: "gelf"
          options:
            gelf-address: "udp://localhost:12201"
            tag: front-nginx