Importing and updating data in Elasticsearch

Question

We have an existing search function that involves data across multiple tables in SQL Server. This causes a heavy load on our DB, so I\'m trying to find a better way to searc

Answer 1

Have you tried changing the config to this:

filter {
  csv {
    columns => ["_id","postal_code","address_1","city","state_code"]
    separator => "|"
  }
}

By naming property_id as _id it should get used during indexing.

Answer 2

Converting from comment:

You can overwrite a document by sending another document with the same ID... but that might be tricky with your previous data, since you'll get randomized IDs by default.

You can set an ID using the output plugin's document_id field, but it takes a literal string, not a field name. To use a field's contents, you could use an sprintf format string, such as %{property_id}.

Something like this, for example:

output {
  elasticsearch {
    ... other settings...
    document_id => "%{property_id}"
  }
}

Answer 3

declaimer - I'm the author of ESL
You can use elasticsearch_loader to load psv files into elasticsearch.
In order to set the _id field you can use --id-field=property_id. for instance:
elasticsearch_loader --index=myindex --type=mytype --id-field=property_id csv --delimiter='|' filename.csv