how to implement the unit or integration tests for logstash configuration?

Question

With the logstash 1.2.1 one can now have conditional to do various stuff. Even the earlier version\'s conf file can get complicated if one is managing many log files and impleme

Answer 1

since logstash has been upgraded and now the command will be something like (give the folder)

/opt/logstash/bin/logstash agent --configtest  -f /etc/logstash/logstash-indexer/conf.d

If you see some warning, but the error message is mixed together, and you didn't know which one have issue. You have to check its file one by one

Answer 2

For a syntax check, there is --configtest:

java -jar logstash.jar agent --configtest --config <yourconfigfile>

To test the logic of the configuration you can write rspec tests. This is an example rspec file to test a haproxy log filter:

require "test_utils"

describe "haproxy logs" do
  extend LogStash::RSpec

  config <<-CONFIG
  filter {
    grok {
       type            => "haproxy"
       add_tag         => [ "HTTP_REQUEST" ]
       pattern         => "%{HAPROXYHTTP}"
    }
    date {
       type            => 'haproxy'
       match           => [ 'accept_date', 'dd/MMM/yyyy:HH:mm:ss.SSS' ]
    }
  }
  CONFIG

  sample({'@message' => '<150>Oct 8 08:46:47 syslog.host.net haproxy[13262]: 10.0.1.2:44799 [08/Oct/2013:08:46:44.256] frontend-name backend-name/server.host.net 0/0/0/1/2 404 1147 - - ---- 0/0/0/0/0 0/0 {client.host.net||||Apache-HttpClient/4.1.2 (java 1. 5)} {text/html;charset=utf-8|||} "GET /app/status HTTP/1.1"',
          '@source_host' => '127.0.0.1',
          '@type' => 'haproxy',
          '@source' => 'tcp://127.0.0.1:60207/',

      }) do

    insist { subject["@fields"]["backend_name"] } == [ "backend-name" ]
    insist { subject["@fields"]["http_request"] } == [ "/app/status" ]
    insist { subject["tags"].include?("HTTP_REQUEST") }
    insist { subject["@timestamp"] } == "2013-10-08T06:46:44.256Z"
    reject { subject["@timestamp"] } == "2013-10-08T06:46:47Z"
  end
end

This will, based on a given filter configuration, run input samples and test if the expected output is produced.

To run the test, save the test as haproxy_spec.rb and run `logstash rspec:

java -jar logstash.jar rspec haproxy_spec.rb

There are lots of spec examples in the Logstash source repository.