Error 1: kibana login error, check the log and find: crypt_r() failed

$ cat /var/log/nginx/error.log
2020/01/28 12:11:38 [crit] 8863#8863: *3 crypt_r() failed (22: Invalid argument), client: 10.20.44.113, server: xx.xx.xx.xx, request: "GET /status HTTP/1.1", host: "xx.xx.xx.xx"

Cause analysis

# When I first created the user, I had already fouled. As a result, the htpasswd file looks like this:
$ cat /etc/nginx/htpasswd.users
kibanaadmin:
kibanaadmin:$apr1$sNJR/eWP$4YZjLMfSka13/UfkLH2.J.

Solution

# After deleting the blank user, everything is OK.
$ sudo nano /etc/nginx/htpasswd.users
$ cat /etc/nginx/htpasswd.users
kibanaadmin:$apr1$sNJR/eWP$4YZjLMfSka13/UfkLH2.J.

Error 2: failed to connect to elasticsearch on the server with filebeat collection log installed

Failed to connect to xx.xx.xx.xx port 9200: Connection refused

# Wrong details
$ sudo filebeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["xx.xx.xx.xx:9200"]'
Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch http://xx.xx.xx.xx:9200: Get http://xx.xx.xx.xx:9200: dial tcp xx.xx.xx.xx:9200: connect: connection refused]

$ curl -XGET 'http://xx.xx.xx.xx:9200/filebeat-*/_search?pretty'
curl: (7) Failed to connect to xx.xx.xx.xx port 9200: Connection refused

Solution: replace the localhost in the elastic search configuration file with IP

$ sudo nano /etc/logstash/conf.d/30-elasticsearch-output.conf
output {
  elasticsearch {
    hosts => ["xx.xx.xx.xx:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

Error reporting 3: curl http://xx.xx.xx.xx:5601 error reporting: Kibana server is not ready yet

Error 4: fail to get the kibana version: http get request to / API / status failures

Error reporting details

sudo filebeat setup -e -E output.logstash.enabled=false -E output.elasticsearch.hosts=['xx.xx.xx.xx:9200'] -E setup.kibana.host=xx.xx.xx.xx:5601

2020-01-29T10:25:38.245+0800    INFO    instance/beat.go:280    Setup Beat: filebeat; Version: 6.8.6
2020-01-29T10:25:38.246+0800    INFO    elasticsearch/client.go:164     Elasticsearch url: http://xx.xx.xx.xx:9200
2020-01-29T10:25:38.246+0800    INFO    [publisher]     pipeline/module.go:110  Beat name: elk
2020-01-29T10:25:38.246+0800    INFO    elasticsearch/client.go:164     Elasticsearch url: http://xx.xx.xx.xx:9200
2020-01-29T10:25:38.278+0800    INFO    elasticsearch/client.go:739     Attempting to connect to Elasticsearch version 6.8.6
2020-01-29T10:25:38.396+0800    INFO    template/load.go:128    Template already exists and will not be overwritten.
2020-01-29T10:25:38.396+0800    INFO    instance/beat.go:889    Template successfully loaded.
Loaded index template
Loading dashboards (Kibana must be running and reachable)
2020-01-29T10:25:38.396+0800    INFO    elasticsearch/client.go:164     Elasticsearch url: http://xx.xx.xx.xx:9200
2020-01-29T10:25:38.398+0800    INFO    elasticsearch/client.go:739     Attempting to connect to Elasticsearch version 6.8.6
2020-01-29T10:25:38.495+0800    INFO    kibana/client.go:118    Kibana url: http://xx.xx.xx.xx:5601
2020-01-29T10:25:38.495+0800    ERROR   instance/beat.go:906    Exiting: fail to create the Kibana loader: Error creating Kibana client: Error creating Kibana client: fail to get the Kibana version: HTTP GET request to /api/status fails: fail to execute the HTTP GET request: Get **http://xx.xx.xx.xx:5601/api/status: dial tcp xx.xx.xx.xx:5601: connect: connection refused. Response: .**
Exiting: fail to create the Kibana loader: Error creating Kibana client: Error creating Kibana client: fail to get the Kibana version: HTTP GET request to /api/status fails: fail to execute the HTTP GET request: Get http://xx.xx.xx.xx:5601/api/status: dial tcp xx.xx.xx.xx:5601: connect: connection refused. Response: .

Solution (error 3 and 4): add the following configuration on the server where kibanna is installed:

$ sudo nano /etc/kibana/kibana.yml
server.port: 5601
server.host: "xx.xx.xx.xx"
elasticsearch.hosts: ["http://xx.xx.xx.xx:9200"]
$ grep ^[a-Z] /etc/kibana/kibana.yml
server.port: 5601
server.host: "xx.xx.xx.xx"
elasticsearch.hosts: ["http://xx.xx.xx.xx:9200"]
# After changing the profile, restart
$ sudo systemctl restart kibana