How to fix ElasticSearch conflicts on the same key when two process writing at the same time

Question

I have multiple processes to write data to ES at the same time, also two processes may write the same key with different values at the same time, it caused the exception as foll

Answer 1

The ES provides the ability to use the retry_on_conflict query parameter.

Specify how many times should the operation be retried when a conflict occurs. Default: 0.

If you have several parallel scripts that can simultaneously work with the same document, you can use this parameter.

For example: You have an index for tweets. And 5 processes that will work with this index. It is possible that all 5 scripts will work with the same document (some tweet). In this case, you can use the ...&retry_on_conflict=6 parameter. Why 6? 5 processes + 1 (plus some legroom). Thus, the ES will try to re-update the document up to 6 times if conflicts occur.