logstash

问题 We are trying to migrate around 3 million records from oracle to Elastic Search using Logstash. We are applying a couple of jdbc_streaming filters as a part of our logstash script, one to load connecting nested objects and another to run a hierarchical query to load data to another nested object in the index. We are able to index 0.4 million records in 24 hours. The total size occupied by .4 million records is around 300MB. We tried multiple approaches to migrate data quickly into elastic

问题 I'm writing a kibana plugin and a logstash pipeline. For my tests, I just wrote a logstash input like that: input { beats { port => 9600 ssl => false ssl_verify_mode => "none" } } But when I try to open a connection with node (code above): invoke = (parameters, id, port, host) => { var fs = require('fs'); console.log(`Sending message in beats, host= ${host}, port= ${port}, message= ${parameters.message}`); var connectionOptions = { host: host, port: port }; var client = lumberjack.client

问题 {:timestamp=>"2017-07-19T15:56:36.517000+0530", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error} {:timestamp=>"2017-07-19T15:56:37.761000+0530", :message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :backtrace=>["/opt/logstash/vendor

问题 I am using a Logstash plugin (logstash-input-rethinkdb). This plugin grabs all the edits in the database and outputs an json object containing the following structure: { "db":"itjobs", "table":"countries", "old_val":null, "new_val":{ "code":"USA3", "country":"USA3", "id":"7c8c9e4e-aa37-48f1-82a5d624cde4a3a0" }, "@version":"1", "@timestamp":"2016-12-19T19:54:08.263Z" } I insert this doc in elasticsearch. But the problem is that in elastic i get the same structure with -> new_val:{code:''} I

问题 I am using a Logstash plugin (logstash-input-rethinkdb). This plugin grabs all the edits in the database and outputs an json object containing the following structure: { "db":"itjobs", "table":"countries", "old_val":null, "new_val":{ "code":"USA3", "country":"USA3", "id":"7c8c9e4e-aa37-48f1-82a5d624cde4a3a0" }, "@version":"1", "@timestamp":"2016-12-19T19:54:08.263Z" } I insert this doc in elasticsearch. But the problem is that in elastic i get the same structure with -> new_val:{code:''} I

问题 I have read that it is possible to assign dynamic names to the indexes like this: elasticsearch { cluster => "logstash" index => "logstash-%{clientid}-%{+YYYY.MM.dd}" } What I am wondering is if it is possible to assign the template dynamically as well: elasticsearch { cluster => "logstash" template => "/etc/logstash/conf.d/%{clientid}-template.json" } Also where does the variable %{clientid} come from? Thanks! 回答1: Full disclosure: I am a Logstash developer at Elastic You cannot dynamically

问题 I intend to have an ELK stack setup where daily JSON inputs get stored in log files created, one for each date. My logstash shall listen to the input via these logs and store it to Elasticsearch at an index corresponding to the date of the log file entry. My logstash-output.conf goes something like: output { elasticsearch { host => localhost cluster => "elasticsearch_prod" index => "test" } } Thus, as for now, all the inputs to logstash get stored at index test of elasticsearch. What I want

问题 set it up like below wget https://artifacts.elastic.co/downloads/logstash/logstash-6.6.2.deb sudo dpkg -i logstash-6.6.2.deb sudo systemctl enable logstash.service sudo systemctl start logstash.service and i added a pipeline script like below input { file { path => "/root/dev/Intuseer-PaaS/backend/airound_sv_logs.log" start_position => "beginning" } } output { stdout {} file { path => "/root/dev/output/output-%{+YYYY-MM-dd}.log" } } the log file likes below timestamp, server_cpu, server

问题 set it up like below wget https://artifacts.elastic.co/downloads/logstash/logstash-6.6.2.deb sudo dpkg -i logstash-6.6.2.deb sudo systemctl enable logstash.service sudo systemctl start logstash.service and i added a pipeline script like below input { file { path => "/root/dev/Intuseer-PaaS/backend/airound_sv_logs.log" start_position => "beginning" } } output { stdout {} file { path => "/root/dev/output/output-%{+YYYY-MM-dd}.log" } } the log file likes below timestamp, server_cpu, server