问题
Context: I have been struggling this whole week to get this stack up and running: filebeat -> kafka -> logstash -> elasticsearch - kibana, each one in its own docker (you will find around 3 or 4 other questions mine here without answer resulted from different tentatives). I have decided to downsize the stack and then move block by block untill i reach a final docker-compose. Then I tried the simplest stack I can imagine to push forward the simplest log I can imagine and I am facing the issue mentioned above in my question.
Issue: I am trying to run straight from command line three docker containers: filebeat, elasticsearch and kibana. When I try to start kibana I get "No living connections". I am following carefully the answer provide in another stackoverflow question. Any clue why I am not able to connect from Kibana container to Elasticsearch container?
Here are all three docker commands:
docker run -d -p 9200:9200 -e "discovery.type=single-node" --volume C:\Dockers\simplest-try\esdata:/usr/share/elasticsearch/data --name elasticsearch_container docker.elastic.co/elasticsearch/elasticsearch:7.5.2
docker run -d --mount type=bind,source=C:\Dockers\simplest-try\filebeat.yml,target=/usr/share/filebeat/filebeat.yml --volume C:\Dockers\simplest-try\mylogs:/mylogs docker.elastic.co/beats/filebeat:7.5.2
docker run -d --name kibana -p 5601:5601 --link elasticsearch_container:elasticsearch_alias -e "ELASTICSEARCH_URL=http://elasticsearch_alias:9200" docker.elastic.co/kibana/kibana:7.5.2
ElasticSearch is up and running:
C:\Dockers\simplest-try>curl localhost:9200
{
"name" : "ffaa2d39a8b2",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "QWYLaAqwSqu76fNwFtZ5AA",
"version" : {
"number" : "7.5.2",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "8bec50e1e0ad29dad5653712cf3bb580cd1afcdf",
"build_date" : "2020-01-15T12:11:52.313576Z",
"build_snapshot" : false,
"lucene_version" : "8.3.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Kibana container console:
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins-system"],"pid":6,"message":"Setting up [15] plugins: [security,licensing,code,timelion,features,spaces,translations,uiActions,newsfeed,inspector,embeddable,advancedUiActions,expressions,eui_utils,data]"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","security"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["warning","plugins","security","config"],"pid":6,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["warning","plugins","security","config"],"pid":6,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","licensing"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","code"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","timelion"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","features"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","spaces"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","translations"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","data"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:41Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nGET http://elasticsearch:9200/_xpack => getaddrinfo ENOTFOUND elasticsearch elasticsearch:9200"}
{"type":"log","@timestamp":"2020-02-06T14:53:42Z","tags":["warning","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/visualizations","message":"Skipping non-plugin directory at /usr/share/kibana/src/legacy/core_plugins/visualizations"}
{"type":"log","@timestamp":"2020-02-06T14:53:42Z","tags":["info","plugins-system"],"pid":6,"message":"Starting [8] plugins: [security,licensing,code,timelion,features,spaces,translations,data]"}
{"type":"log","@timestamp":"2020-02-06T14:53:42Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-02-06T14:53:42Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-02-06T14:53:42Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch for the [data] cluster. Error: No Living connections"}
{"type":"log","@timestamp":"2020-02-06T14:53:43Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET http://elasticsearch:9200/.kibana => getaddrinfo ENOTFOUND elasticsearch elasticsearch:9200"}
{"type":"log","@timestamp":"2020-02-06T14:53:43Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET http://elasticsearch:9200/.kibana_task_manager => getaddrinfo ENOTFOUND elasticsearch elasticsearch:9200"}
{"type":"log","@timestamp":"2020-02-06T14:53:44Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-02-06T14:53:44Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-02-06T14:53:44Z","tags":["warning","migrations"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: No Living connections"}
Although not straigt to my question title, here are details about Filebeat: Filebeat try to harverst my log files
2020-02-06T14:32:23.782Z INFO crawler/crawler.go:72 Loading Inputs: 1
2020-02-06T14:32:23.782Z INFO log/input.go:152 Configured paths: [/mylogs/*.log]
2020-02-06T14:32:23.782Z INFO input/input.go:114 Starting input of type: log; ID: 4094557846902174710
2020-02-06T14:32:23.782Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2020-02-06T14:32:23.788Z INFO log/harvester.go:251 Harvester started for file: /mylogs/y.log
2020-02-06T14:32:23.790Z INFO log/harvester.go:251 Harvester started for file: /mylogs/x.log
filebeat.yml
filebeat.inputs:
- type: log
paths:
- '/mylogs/*.log'
json.message_key: log
json.keys_under_root: true
processors:
- add_docker_metadata: ~
output.elasticsearch:
hosts: ["localhost:9200"]
*** edited
logs after Ibexit's suggestion
2020-02-12T21:33:03.575Z INFO instance/beat.go:610 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-02-12T21:33:03.588Z INFO instance/beat.go:618 Beat ID: d0c71c07-23e0-44e5-b497-195ee9552fe8
2020-02-12T21:33:03.588Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2020-02-12T21:33:03.588Z INFO [beat] instance/beat.go:941 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "d0c71c07-23e0-44e5-b497-195ee9552fe8"}}}
2020-02-12T21:33:03.588Z INFO [beat] instance/beat.go:950 Build info {"system_info": {"build": {"commit": "a9c141434cd6b25d7a74a9c770be6b70643dc767", "libbeat": "7.5.2", "time": "2020-01-15T11:13:22.000Z", "version": "7.5.2"}}}
2020-02-12T21:33:03.588Z INFO [beat] instance/beat.go:953 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.12.12"}}}
2020-02-12T21:33:03.590Z INFO [beat] instance/beat.go:957 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-02-12T20:32:39Z","containerized":true,"name":"fcfaea4080e7","ip":["127.0.0.1/8","172.17.0.3/16"],"kernel_version":"4.19.76-linuxkit","mac":["02:42:ac:11:00:03"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-02-12T21:33:03.590Z INFO [beat] instance/beat.go:986 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-02-12T21:33:02.690Z"}}}
2020-02-12T21:33:03.590Z INFO instance/beat.go:297 Setup Beat: filebeat; Version: 7.5.2
2020-02-12T21:33:03.590Z INFO [index-management] idxmgmt/std.go:182 Set output.elasticsearch.index to 'filebeat-7.5.2' as ILM is enabled.
2020-02-12T21:33:03.591Z INFO elasticsearch/client.go:171 Elasticsearch url: http://elasticsearch:9200
2020-02-12T21:33:03.591Z INFO [publisher] pipeline/module.go:97 Beat name: fcfaea4080e7
2020-02-12T21:33:03.593Z INFO instance/beat.go:429 filebeat start running.
2020-02-12T21:33:03.593Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-02-12T21:33:03.594Z INFO registrar/migrate.go:104 No registry home found. Create: /usr/share/filebeat/data/registry/filebeat
2020-02-12T21:33:03.594Z INFO registrar/migrate.go:112 Initialize registry meta file
2020-02-12T21:33:03.600Z INFO registrar/registrar.go:108 No registry file found under: /usr/share/filebeat/data/registry/filebeat/data.json. Creating a new registry file.
2020-02-12T21:33:03.611Z INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-02-12T21:33:03.611Z INFO registrar/registrar.go:152 States Loaded from registrar: 0
2020-02-12T21:33:03.612Z INFO crawler/crawler.go:72 Loading Inputs: 1
2020-02-12T21:33:03.612Z INFO log/input.go:152 Configured paths: [/mylogs/*.log]
2020-02-12T21:33:03.612Z INFO input/input.go:114 Starting input of type: log; ID: 4094557846902174710
2020-02-12T21:33:03.612Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2020-02-12T21:33:03.640Z INFO log/harvester.go:251 Harvester started for file: /mylogs/b.log
2020-02-12T21:33:03.640Z ERROR readjson/json.go:52 Error decoding JSON: invalid character '\'' looking for beginning of object key string
2020-02-12T21:33:03.642Z INFO log/harvester.go:251 Harvester started for file: /mylogs/c.log
2020-02-12T21:33:03.644Z INFO log/harvester.go:251 Harvester started for file: /mylogs/w.log
2020-02-12T21:33:03.645Z ERROR readjson/json.go:52 Error decoding JSON: invalid character 'q' looking for beginning of value
2020-02-12T21:33:03.645Z INFO log/harvester.go:251 Harvester started for file: /mylogs/x.log
2020-02-12T21:33:03.652Z INFO log/harvester.go:251 Harvester started for file: /mylogs/y.log
2020-02-12T21:33:04.654Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(http://elasticsearch:9200))
2020-02-12T21:33:04.684Z INFO elasticsearch/client.go:753 Attempting to connect to Elasticsearch version 7.5.2
2020-02-12T21:33:04.720Z INFO [index-management] idxmgmt/std.go:256 Auto ILM enable success.
2020-02-12T21:33:04.724Z INFO [index-management.ilm] ilm/std.go:138 do not generate ilm policy: exists=true, overwrite=false
2020-02-12T21:33:04.724Z INFO [index-management] idxmgmt/std.go:269 ILM policy successfully loaded.
2020-02-12T21:33:04.725Z INFO [index-management] idxmgmt/std.go:408 Set setup.template.name to '{filebeat-7.5.2 {now/d}-000001}' as ILM is enabled.
2020-02-12T21:33:04.725Z INFO [index-management] idxmgmt/std.go:413 Set setup.template.pattern to 'filebeat-7.5.2-*' as ILM is enabled.
2020-02-12T21:33:04.725Z INFO [index-management] idxmgmt/std.go:447 Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.5.2 {now/d}-000001} as ILM is enabled.
2020-02-12T21:33:04.725Z INFO [index-management] idxmgmt/std.go:451 Set settings.index.lifecycle.name in template to {filebeat-7.5.2 {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-02-12T21:33:04.730Z INFO template/load.go:89 Template filebeat-7.5.2 already exists and will not be overwritten.
2020-02-12T21:33:04.730Z INFO [index-management] idxmgmt/std.go:293 Loaded index template.
2020-02-12T21:33:04.734Z INFO [index-management] idxmgmt/std.go:304 Write alias successfully generated.
2020-02-12T21:33:04.736Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch(http://elasticsearch:9200)) established
2020-02-12T21:33:33.595Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":50,"time":{"ms":50}},"total":{"ticks":100,"time":{"ms":107},"value":100},"user":{"ticks":50,"time":{"ms":57}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":11},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":30060}},"memstats":{"gc_next":8351264,"memory_alloc":4760176,"memory_total":12037984,"rss":43970560},"runtime":{"goroutines":42}},"filebeat":{"events":{"added":8,"done":8},"harvester":{"open_files":5,"running":5,"started":5}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":3,"batches":1,"total":3},"read":{"bytes":2942},"type":"elasticsearch","write":{"bytes":2545}},"pipeline":{"clients":1,"events":{"active":0,"filtered":5,"published":3,"retry":3,"total":8},"queue":{"acked":3}}},"registrar":{"states":{"current":5,"update":8},"writes":{"success":7,"total":7}},"system":{"cpu":{"cores":2},"load":{"1":0.02,"15":0.08,"5":0.1,"norm":{"1":0.01,"15":0.04,"5":0.05}}}}}}
2020-02-12T21:33:58.657Z ERROR readjson/json.go:52 Error decoding JSON: invalid character 'E' looking for beginning of value
2020-02-12T21:33:58.657Z ERROR readjson/json.go:52 Error decoding JSON: invalid character 'a' looking for beginning of value
2020-02-12T21:34:03.596Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":60,"time":{"ms":13}},"total":{"ticks":120,"time":{"ms":16},"value":120},"user":{"ticks":60,"time":{"ms":3}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":11},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":60059}},"memstats":{"gc_next":8351264,"memory_alloc":5345000,"memory_total":12622808},"runtime":{"goroutines":42}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"open_files":5,"running":5}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":2,"batches":1,"total":2},"read":{"bytes":351},"write":{"bytes":1062}},"pipeline":{"clients":1,"events":{"active":0,"published":2,"total":2},"queue":{"acked":2}}},"registrar":{"states":{"current":5,"update":2},"writes":{"success":1,"total":1}},"system":{"load":{"1":0.01,"15":0.08,"5":0.09,"norm":{"1":0.005,"15":0.04,"5":0.045}}}}}}
2020-02-12T21:34:33.599Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":70,"time":{"ms":10}},"total":{"ticks":130,"time":{"ms":14},"value":130},"user":{"ticks":60,"time":{"ms":4}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":11},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":90059}},"memstats":{"gc_next":8351264,"memory_alloc":5714936,"memory_total":12992744,"rss":380928},"runtime":{"goroutines":42}},"filebeat":{"harvester":{"open_files":5,"running":5}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":5}},"system":{"load":{"1":0.07,"15":0.08,"5":0.1,"norm":{"1":0.035,"15":0.04,"5":0.05}}}}}}
2020-02-12T21:34:33.686Z INFO log/harvester.go:251 Harvester started for file: /mylogs/d.log
2020-02-12T21:35:03.597Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":16}},"total":{"ticks":140,"time":{"ms":21},"value":140},"user":{"ticks":60,"time":{"ms":5}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":120059}},"memstats":{"gc_next":8351264,"memory_alloc":6130552,"memory_total":13408360},"runtime":{"goroutines":46}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":6,"running":6,"started":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":6,"update":1},"writes":{"success":1,"total":1}},"system":{"load":{"1":0.15,"15":0.09,"5":0.12,"norm":{"1":0.075,"15":0.045,"5":0.06}}}}}}
2020-02-12T21:35:33.596Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":100,"time":{"ms":14}},"total":{"ticks":170,"time":{"ms":23},"value":170},"user":{"ticks":70,"time":{"ms":9}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":150060}},"memstats":{"gc_next":7948720,"memory_alloc":4110408,"memory_total":13866968},"runtime":{"goroutines":46}},"filebeat":{"harvester":{"open_files":6,"running":6}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":6}},"system":{"load":{"1":0.09,"15":0.08,"5":0.11,"norm":{"1":0.045,"15":0.04,"5":0.055}}}}}}
2020-02-12T21:36:03.597Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":110,"time":{"ms":7}},"total":{"ticks":190,"time":{"ms":9},"value":190},"user":{"ticks":80,"time":{"ms":2}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":180059}},"memstats":{"gc_next":7948720,"memory_alloc":4399584,"memory_total":14156144},"runtime":{"goroutines":46}},"filebeat":{"harvester":{"open_files":6,"running":6}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":6}},"system":{"load":{"1":0.38,"15":0.11,"5":0.18,"norm":{"1":0.19,"15":0.055,"5":0.09}}}}}}
2020-02-12T21:36:33.596Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":120,"time":{"ms":11}},"total":{"ticks":200,"time":{"ms":15},"value":200},"user":{"ticks":80,"time":{"ms":4}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":210059}},"memstats":{"gc_next":7948720,"memory_alloc":4776320,"memory_total":14532880},"runtime":{"goroutines":46}},"filebeat":{"harvester":{"open_files":6,"running":6}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":6}},"system":{"load":{"1":0.23,"15":0.1,"5":0.16,"norm":{"1":0.115,"15":0.05,"5":0.08}}}}}}
2020-02-12T21:37:03.600Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":120,"time":{"ms":9}},"total":{"ticks":210,"time":{"ms":16},"value":210},"user":{"ticks":90,"time":{"ms":7}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":240059}},"memstats":{"gc_next":7948720,"memory_alloc":5142416,"memory_total":14898976},"runtime":{"goroutines":46}},"filebeat":{"harvester":{"open_files":6,"running":6}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":6}},"system":{"load":{"1":0.14,"15":0.1,"5":0.14,"norm":{"1":0.07,"15":0.05,"5":0.07}}}}}}
2020-02-12T21:37:33.596Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":{"ms":12}},"total":{"ticks":240,"time":{"ms":24},"value":240},"user":{"ticks":100,"time":{"ms":12}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":270060}},"memstats":{"gc_next":7946160,"memory_alloc":4111832,"memory_total":15348288},"runtime":{"goroutines":46}},"filebeat":{"harvester":{"open_files":6,"running":6}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":6}},"system":{"load":{"1":0.08,"15":0.09,"5":0.13,"norm":{"1":0.04,"15":0.045,"5":0.065}}}}}}
2020-02-12T21:38:03.596Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":150,"time":{"ms":11}},"total":{"ticks":250,"time":{"ms":12},"value":250},"user":{"ticks":100,"time":{"ms":1}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":300060}},"memstats":{"gc_next":7946160,"memory_alloc":4489960,"memory_total":15726416},"runtime":{"goroutines":46}},"filebeat":{"harvester":{"open_files":6,"running":6}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":6}},"system":{"load":{"1":0.1,"15":0.09,"5":0.13,"norm":{"1":0.05,"15":0.045,"5":0.065}}}}}}
2020-02-12T21:38:08.676Z INFO log/harvester.go:276 File is inactive: /mylogs/w.log. Closing because close_inactive of 5m0s reached.
2020-02-12T21:38:08.676Z INFO log/harvester.go:276 File is inactive: /mylogs/c.log. Closing because close_inactive of 5m0s reached.
2020-02-12T21:38:08.678Z INFO log/harvester.go:276 File is inactive: /mylogs/b.log. Closing because close_inactive of 5m0s reached.
2020-02-12T21:38:08.678Z INFO log/harvester.go:276 File is inactive: /mylogs/y.log. Closing because close_inactive of 5m0s reached.
2020-02-12T21:38:13.706Z INFO log/harvester.go:251 Harvester started for file: /mylogs/y.log
2020-02-12T21:38:33.594Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":150,"time":{"ms":5}},"total":{"ticks":250,"time":{"ms":9},"value":250},"user":{"ticks":100,"time":{"ms":4}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":330059}},"memstats":{"gc_next":7946160,"memory_alloc":5014240,"memory_total":16250696},"runtime":{"goroutines":34}},"filebeat":{"events":{"added":5,"done":5},"harvester":{"closed":4,"open_files":3,"running":3,"started":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":5,"total":5}}},"registrar":{"states":{"current":6,"update":5},"writes":{"success":5,"total":5}},"system":{"load":{"1":0.88,"15":0.15,"5":0.31,"norm":{"1":0.44,"15":0.075,"5":0.155}}}}}}
2020-02-12T21:39:03.595Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":160,"time":{"ms":6}},"total":{"ticks":270,"time":{"ms":8},"value":270},"user":{"ticks":110,"time":{"ms":2}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":360059}},"memstats":{"gc_next":7946160,"memory_alloc":5284712,"memory_total":16521168},"runtime":{"goroutines":34}},"filebeat":{"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":6}},"system":{"load":{"1":0.68,"15":0.16,"5":0.31,"norm":{"1":0.34,"15":0.08,"5":0.155}}}}}}
2020-02-12T21:39:03.676Z INFO log/harvester.go:276 File is inactive: /mylogs/x.log. Closing because close_inactive of 5m0s reached.
2020-02-12T21:39:33.596Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":160,"time":{"ms":5}},"total":{"ticks":270,"time":{"ms":12},"value":270},"user":{"ticks":110,"time":{"ms":7}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":390059}},"memstats":{"gc_next":7666032,"memory_alloc":3879448,"memory_total":16793464},"runtime":{"goroutines":30}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"closed":1,"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":6,"update":1},"writes":{"success":1,"total":1}},"system":{"load":{"1":0.48,"15":0.16,"5":0.3,"norm":{"1":0.24,"15":0.08,"5":0.15}}}}}}
2020-02-12T21:39:38.705Z INFO log/harvester.go:276 File is inactive: /mylogs/d.log. Closing because close_inactive of 5m0s reached.
2020-02-12T21:39:43.714Z INFO log/harvester.go:251 Harvester started for file: /mylogs/d.log
2020-02-12T21:39:43.715Z ERROR readjson/json.go:52 Error decoding JSON: EOF
2020-02-12T21:39:49.724Z INFO log/harvester.go:264 File was truncated. Begin reading file from offset 0: /mylogs/d.log
2020-02-12T21:39:53.720Z INFO log/harvester.go:251 Harvester started for file: /mylogs/d.log
2020-02-12T21:39:53.721Z ERROR readjson/json.go:52 Error decoding JSON: EOF
2020-02-12T21:40:03.597Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":190,"time":{"ms":30}},"total":{"ticks":320,"time":{"ms":46},"value":320},"user":{"ticks":130,"time":{"ms":16}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":420059}},"memstats":{"gc_next":7666032,"memory_alloc":4930512,"memory_total":17844528},"runtime":{"goroutines":30}},"filebeat":{"events":{"added":8,"done":8},"harvester":{"closed":2,"open_files":2,"running":2,"started":2},"input":{"log":{"files":{"truncated":1}}}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":4,"batches":2,"total":4},"read":{"bytes":702},"write":{"bytes":2270}},"pipeline":{"clients":1,"events":{"active":0,"filtered":4,"published":4,"total":8},"queue":{"acked":4}}},"registrar":{"states":{"current":6,"update":8},"writes":{"success":6,"total":6}},"system":{"load":{"1":0.59,"15":0.17,"5":0.33,"norm":{"1":0.295,"15":0.085,"5":0.165}}}}}}
回答1:
The Problem is that the three container are separated in terms of networking from each other and/or misconfigured. Let us discuss what is actualy happening and how to fix it:
1. Elasticsearch
You are starting an elasticsearch container named elasticsearch_container:
docker run -d -p 9200:9200 -e "discovery.type=single-node" --volume C:\Dockers\simplest-try\esdata:/usr/share/elasticsearch/data --name elasticsearch_container docker.elastic.co/elasticsearch/elasticsearch:7.5.2
So far, so good.
2. Filebeat
As mentioned at the beginning, the containers are separated from each other. In order to make elasticsearch visible for filebeat, you need to create a link:
docker run -d --link elasticsearch_container:elasticsearch --mount type=bind,source=C:\Dockers\simplest-try\filebeat.yml,target=/usr/share/filebeat/filebeat.yml --volume C:\Dockers\simplest-try\mylogs:/mylogs docker.elastic.co/beats/filebeat:7.5.2
Please note the container link: --link elasticsearch_container:elasticsearch which is the key here. Now, as the elasticsearch_container is visible to filebeat under the name elasticsearch, we need to change the filebeat.yml in that way:
output.elasticsearch:
hosts: ["http://elasticsearch:9200"]
Using localhost here is meant from the perspective of the filebeat container, which is unaware of the docker host. So localhost within the filebeat container adresses the filebeat container itself. But with the configuration change above, we changed it to the name of the linked elasticsearch container, what should do the trick.
3. Kibana
Kibana is complaining about missing connection to elasticsearch:
Unable to revive connection: http://elasticsearch:9200
Here it's the same case as for filebeat: elasticsearch is not visible to the kibana container under the name elasticsearch but elasticsearch_alias. Additionally, ELASTICSEARCH_URL is not an expected configuration in the version you are using. elasticsearch.hosts is the correct setting and defaults to http://elasticsearch:9200. And this is the root of the error message: kibana is not recognising ELASTICSEARCH_URL, falls back to the default value and fails because elasticsearch_container is linked as elasticsearch_alias and not as elasticsearch. Fixing this is easy, as we need just to remove ELASTICSEARCH_URL and let kibana fall back to the default. To make elasticsearch visible to kibana, we just apply the same link as we did for filebeat already:
docker run -d --name kibana -p 5601:5601 --link elasticsearch_container:elasticsearch docker.elastic.co/kibana/kibana:7.5.2
Important: Please dispose (stop & remove) the old container instances as they are claiming the container names before executing the discussed changes.
来源：https://stackoverflow.com/questions/60120210/kibana-not-connecting-to-elasticsearch-warning-elasticsearch-admin-pid