How to generate csv response in elasticsearch?

Question

As we know, the rest apis of Elasticsearch returns json response.But, I need CSV responses from those apis.

I

Answer 1

This is somewhat difficult, because inherently - JSON is a hierarchical data structure, and CSV is not.

There isn't a trivial way of reducing one to the other as a result - anything you do will be custom.

However you can do something like:

#!/usr/bin/env perl

use strict;
use warnings;

use LWP;
use JSON;

my $url =
  'http://localhost:9200/index-name/path/AVm7dsU_mwKGPn0NRXkK';

my $agent    = LWP::UserAgent->new;
my $response = $agent->get($url);

if ( $response->code ) {
   my $json = from_json( $response->content );
   my @fields = sort keys %{ $json->{_source} };

   #column headings
   print join ",", @fields, "\n";
   #column values
   print join ",", @{ $json->{_source} }{@fields}, "\n";
}

It's a bit crude, and assumes that with _source there's a flat key-value relationship. With multiple records you'd need to wrap it in a loop to print multiple - this is just an example with a single document.

It would be better - if at all possible - to change whatever is wanting the CSV, to handle a multi dimensional data format in the first place.

Answer 2

If you're open to use Logstash, then you can very easily do this with an elasticsearch input making the query and then a csv output for dumping the data into a CSV file. It'd look like this:

input {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "your_index"
    query => '{"query": {"match_all": {}}}'
  }
}
output {
  csv {
    fields => ["field1", "field2", "field3"]
    path => "/path/to/file.csv"
  }
}

UPDATE

If you need to invoke this dynamically, you could generate this logstash configuration dynamically based on a query that you'd give as input to the shell script:

#!/bin/sh

if [ -z "$LOGSTASH_HOME" ]; then
    echo "WARNING: The LOGSTASH_HOME environment variable is not set!"
    exit 0
fi

LS_CONF="input {
   elasticsearch {
     hosts => [\"localhost:9200\"]
     index => 'megacorp'
     query => '{\"query\":{\"query_string\": {\"query\": \"$1\"}}}'
   }
}
output {
   csv {
     fields => [$2]
     path => \"/path/to/file.csv\"
   }
}"

$LOGSTASH_HOME/bin/logstash -e "$LS_CONF"

Then you can invoke that script with the query my_field:123456 like this

./es_to_csv.sh "my_field:123456" "field1,field2,field3"

This will have the same effect as calling {{elasticUrl}}/_search?q=my_field:123456 and produce a CSV file with the columns field1,field2,field3