Do Elasticsearch CRUD need refresh?

Question

I need to sync a RDBS data with Elasticsearch. The common approach to achieve this is applying changes on the RDBS and then to use a message queue (or a table used to ETL) t

Answer 1

Short answer:

You don't need a refresh. It will be consistent means operations are executed in order. ES makes sure always latest request succeeds. And it makes the changes persistent every index/update/delete request.

In case, there are two write requests received at different network partition for an ID and later one succeeds first, then earlier one will not be updated as consistency is achieved by versioning. Latest version data always succeeds.

Long answer:

You need to look at many concepts like translog, fsync, consistency at ES, 'optimistic concurrency control', versioning, partitioning, availability.

ES achieves consistency using versioning. So when you sent index/update/delete requests it does the following things at high level.

1. Writes it to translog
2. Makes it persistent - there is a default interval property. When that interval elapses or after every index/delete/update operation
3. Sends the request to the node
4. The node which received the request identifies the leader of the partition where the data belongs to.
5. Partition-leader-node writes the data and forwards to other replica-nodes where this partition should be replicated.
6. Once all are acknowledged, return the status to the client via the initial node-which-received-the-request.

There are many concepts/algorithms in this to make it powerful distributed system.