Built a set of elk by myself 7.x Version environment found that using filebeat to collect nginx or apapche could not customize the directory log directory. If you customize the log directory, it would not be segmented. If you use the default log location, it would be segmented

The solution is hereby recorded

Default load path

[root@kangcw error]# /etc/init.d/filebeat status
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-05-18 21:54:15 CST; 3s ago
     Docs: https://www.elastic.co/products/beats/filebeat
 Main PID: 7425 (filebeat)
   CGroup: /system.slice/filebeat.service
           └─7425 /usr/share/filebeat/bin/filebeat -environment systemd -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebe...

May 18 21:54:15 kangcw filebeat[7425]: 2020-05-18T21:54:15.415+0800        INFO        beater/crawler.go:105        Loading and starting Inputs completed. Enabled inputs: 0
May 18 21:54:15 kangcw filebeat[7425]: 2020-05-18T21:54:15.415+0800        INFO        cfgfile/reload.go:175        Config reloader started
May 18 21:54:15 kangcw filebeat[7425]: 2020-05-18T21:54:15.419+0800        INFO        log/input.go:152        Configured paths: [/var/log/nginx/access.log*]  ##here
May 18 21:54:15 kangcw filebeat[7425]: 2020-05-18T21:54:15.419+0800        INFO        log/input.go:152        Configured paths: [/var/log/nginx/error.log*]   ##here
May 18 21:54:15 kangcw filebeat[7425]: 2020-05-18T21:54:15.420+0800        INFO        eslegclient/connection.go:84        elasticsearch url: http://admin.sxdsms.com:9200
May 18 21:54:15 kangcw filebeat[7425]: 2020-05-18T21:54:15.451+0800        INFO        [esclientleg]        eslegclient/connection.go:263        Attempting to connect to Elas...ersion 7.7.0
May 18 21:54:15 kangcw filebeat[7425]: 2020-05-18T21:54:15.478+0800        INFO        input/input.go:114        Starting input of type: log; ID: 2981948324353935418
May 18 21:54:15 kangcw filebeat[7425]: 2020-05-18T21:54:15.478+0800        INFO        input/input.go:114        Starting input of type: log; ID: 16682563900439131229
May 18 21:54:15 kangcw filebeat[7425]: 2020-05-18T21:54:15.478+0800        INFO        cfgfile/reload.go:235        Loading of config files completed.
May 18 21:54:18 kangcw filebeat[7425]: 2020-05-18T21:54:18.402+0800        INFO        [add_cloud_metadata]        add_cloud_metadata/add_cloud_metadata.go:89        add_clou...ot detected.
Hint: Some lines were ellipsized, use -l to show in full.

Custom path location required

[root@kangcw nginx]# pwd
/usr/share/filebeat/module/nginx
[root@kangcw nginx]# tree .
.
|-- access
|   |-- config
|   |   `-- nginx-access.yml
|   |-- ingest
|   |   `-- default.json
|   `-- manifest.yml
|-- error
|   |-- config
|   |   `-- nginx-error.yml
|   |-- ingest
|   |   `-- pipeline.json
|   `-- manifest.yml
|-- ingress_controller
|   |-- config
|   |   `-- ingress_controller.yml
|   |-- ingest
|   |   `-- pipeline.json
|   `-- manifest.yml
`-- module.yml

9 directories, 10 files

The files to be modified are the access directory of / usr/share/filebeat/module/nginx and the manifest file of error directory in this path manifest.yml

Amend to read

access

[root@kangcw nginx]# cat access/manifest.yml 
module_version: "1.0"

var:
  - name: paths
    default:
 #     - /var/log/nginx/access.log*   #prohibit
      - /www/wwwlogs/dzgk.access.log   #Custom path
    os.darwin:
      - /usr/local/var/log/nginx/access.log*
    os.windows:
      - c:/programdata/nginx/logs/*access.log*

ingest_pipeline: ingest/default.json
. . . . slightly


error
[root@kangcw nginx]# cat error/manifest.yml 
module_version: "1.0"

var:
  - name: paths
    default:
#      - /var/log/nginx/error.log*  #prohibit
      - /www/wwwlogs/dzgk.error.log  #Custom path
    os.darwin:
      - /usr/local/var/log/nginx/error.log*
    os.windows:
      - c:/programdata/nginx/logs/error.log*

ingest_pipeline: ingest/pipeline.json
input: config/nginx-error.yml

At this time, filebeat will call nginx module to automatically parse the log, and the log format in nginx configuration file will not be converted

Of course, this is not omnipotent. If you can't parse it, you have to build your own logstash and grok to parse it

I changed the configuration of apache

Test whether receipt is received by apache module of kibana data source

View apache log dashboard

Data is up

Record my filebeat configuration

[root@kangcw nginx]# cat /etc/filebeat/filebeat.yml |grep -v "#\|^$"
filebeat.inputs:
- type: log
  enabled: false   #The custom path is not activated here. If the log is activated, it will not be automatically segmented
  paths:
    - /www/wwwlogs/*.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
  host: "admin.sxdsms.com:5601"
output.elasticsearch:
  hosts: ["admin.sxdsms.com:9200"]
  username: "elastic"
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

See the effect