Why is there inconsistency in Kafka's ordering guarantees when using Idempotent Producer?

Question

I am using Kafka 1.0.1 in my application and I have started using the Idempotent Producer feature that was introduced in 0.11, and I\'ve having trouble understanding the orderin

Answer 1

The ordering is guaranteed when enabling idempotence in the KafkaProducer.

Even if you have max.in.flight.requests.per.connection larger or equal to 5 an idempotent KafkaProducer will still ensure the ordering within a TopicPartition. The description on the "retries" with the relation to the max.in.flight is only applicable if idempotence is disabled.

An idempotent KafkaProducer uses an internal incrementing sequence number to ensure the ordering.

Details are given in KAFKA-5494 and the corresponding documentation on Design for max.in.flight > 1 with idempotence enabled. where the steps 5 and 6 will clarify your question:

With the above assumptions in place, the solution is as follows:

1. We keep track of the last acknowledged sequence sent to a partition. This is updated on every successful ack and thus should always be increasing.

2. We keep track of the next sequence for a batch bound for a given partition.

3. We assigned the nextSequence when the batch is drained. We also increment the nextSequence by the record count of a batch.

4. If a produce request succeeds, we set the last ack’d sequence to be the last sequence of the batch.

5. If a produce request fails, succeeding in flight batches will also fail with an OutOfOrderSequenceException.

6. As such, if the sequence number of a batch is not the successor of the last ack’d sequence, and if it fails with an OutOfOrderSequenceException, we consider this to be retriable.

7. When a batch is requeued, we erase the producer id and sequence number before inserting it in the queue.

8. When the first inflight batch fails (for whatever reason), we reset the nextSequence to be the lastAckdSequence + 1.

9. Thus if a batch fails fatally, the sequence numbers of succeeding batches will be different on the retry. This is fine, because the previous failure was an OutOfSequence exception, which categorically means that the request was rejected.

"[...] if my producer is configured with max.in.flight.requests.per.connection = 5 and let's say that the second request got the out of order exception, what happens to all of the following requests that are already in flight? will this mean that I am for sure out of order?"

As mentioned in step 5 and 6 all succeeding in-flight requests will also fail with an retriable OutOfOrderSequenceExcpetion. As retries are greater than 0 the idempotent KafkaProducer will be able to keep the order.