logstash http_poller first URL request's response should be input to second URL's request param

Question

I have two URLs (due to security concern i will explain by using dummy)

 a> https://xyz.company.com/ui/api/token
 b> https://xyz.company.com/request/t         


        

Answer 1

Yes, it's possible with a mix of an http_poller input and an http output.

Here is the config I came up with:

input {
   # 1. trigger new token requests every hour
   http_poller {
     urls => {
       token => "https://xyz.company.com/ui/api/token"
     }
     interval => 3600
     add_field => {"token" => "%{message}"}
   }
}
filter {
}
output {
   # 2. call the API
   http {
     http_method => "get"
     url => "https://xyz.company.com/request/transaction?date=2016-01-21&token=%{token}"
   }
}

UPDATE

If you want to be able to get the content of the API call and store it in ES, you need a hybrid solution. You need to set up a cron that will call a script that runs the two HTTP calls and stores the results in a file and then you can let logstash tail that file and forward the results to ES.

Shell script to put on cron:

#!/bin/sh

# 1. Get the token
TOKEN=$(curl -s -XGET https://xyz.company.com/ui/api/token)

# 2. Call the API with the token and append JSON to file
curl -s -XGET "https://xyz.company.com/request/transaction?date=2016-01-21&token=$TOKEN" >> api_calls.log

The above script can be set on cron using crontab (or similar), there are plenty of examples out there on how to achieve this.

Then the logstash config can be very simple. It just needs to tail the api_calls.log file and send the document to ES

input {
    file {
        path => "api_calls.log"
        start_position => "beginning"
    }
}
filter {
    json {
        source => "message"
    }
}
output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "my_index"
        document_type" => "my_type"
    }
    stdout {
        codec => "rubydebug"
    }
}