logstash-configuration

You know how there is a Ruby filter for Logstash which enables me to write code in Ruby and it is usually included in the config file as follows filter { ruby { code => "...." } } Now I have two Jar files that I would like to include in my filter so that the input I have can be processed according to the operations I have in these Jar files. However, I cannot (apparently) include the Jar file in the ruby code. I've been looking for a solution. So to answer this, I found this wonderful tutorial from Elastc.co: Shows the steps to create a new gem and use it as a filter for Logstash later on.

I understand that logstash now supports environment variables in config, as described here . But I can't seem to get it working when running logstash as a service. I am on ubuntu 14.04, logstash 1:2.3.3-1, and I launch logstash with sudo service logstash start . A final twist is that I am including logstash in a docker container, and I do NOT want to hardcode the variable value in my Dockerfile, I want it to ultimately be sourced from the command line when I launch the container, e.g. docker run -e ES_HOST='my_host' ...etc... . This is really the main reason I want to use environment variables

I am new to logstash and desparate to setup ELK for one of the usecase. I have found this question relevent to mine Why won't Logstash multiline merge lines based on grok'd field? If multiline filter do not merge lines on grok fields then how do I merge line 2 and 10 from the below log sample? Please help. Using grok patterns I have created a field 'id' which holds the value 715. Line1 - 5/08/06 00:10:35.348 [BaseAsyncApi] [qtp19303632-51]: INFO: [714] CMDC flowcxt=[55c2a5fbe4b0201c2be31e35] method=contentdetail uri=http://10.126.44.161:5600/cmdc/content/programid%3A%2F%2F317977349~programid

Currently I use redis -> s3 -> elastic search -> kibana stack to pipe and visualise my logs. But due to large volume of data in elastic search I can retain logs upto 7 days. I want to bring kafka cluster in this stack and retain logs for more number of days. I am thinking of following stack. app nodes piping logs to kafka -> kafka cluster -> elastics search cluster -> kibana How can I use kafka to retain logs for more number of days? Looking through the Apache Kafka broker configs , there are two properties that determine when a log will get deleted. One by time and the other by space. log

Upon restarting Logstash, at times observed that Logstash duplicates the log events. Was wondering as to what would be the right way to apply start_position , sincedb_path , sincedb_write_interval configuration options. What happens when there are multiple files in the same location as in my example below /home/tom/testData/*.log What happens when the file rotation occurs like for example the XXX.log file is renamed to XXX-<date>.log and a new XXX.log file is created. In this case name doesn't change, but the inode changes. Would highly appreciate if anyone can throw some light on this. input

So I have log messages of the format : [INFO] <blah.blah> 2016-06-27 21:41:38,263 some text [INFO] <blah.blah> 2016-06-28 18:41:38,262 some other text Now I want to drop all logs that does not contain a specific string "xyz" and keep all the rest. I also want to index timestamp. grokdebug is not helping much. This is my attempt : input { file { path => "/Users/username/Desktop/validateLogconf/logs/*" start_position => "beginning" } } filter { grok { match => { "message" => '%{SYSLOG5424SD:loglevel} <%{JAVACLASS:job}> %{GREEDYDATA:content}' } } date { match => [ "Date", "YYYY-mm-dd HH:mm:ss" ]