问题
With the last version of Kafka 0.11.0.0 the Apache team is introducing idempotent producer and transactions. Is It possible to guarantee that an entire set of messages (for example 1 million) we want to log, will be committed only at the end? I would like that, if for example the Producers loose the connection with the brokers and cannot restabilish it, no messages will be seen by the consumers. Is it possible?
回答1:
Yes this is possible using Transactions in your producer. You start a transaction, publish all your messages, and then commit the transaction. All the messages are written to Kafka one at a time but consumers in the new READ_COMMITTED mode will only see the messages after the transaction is committed by the producer and a special transaction marker is added to the Kafka commit log.
Consumers not in READ_COMMITTED mode can see the messages as they are written individually even though they may not yet (or ever) be committed.
There is a limit to how long an open transaction can stay uncommitted so eventually if the producer dies and does not explicitly end the transaction it will timeout and rollback and READ_COMMITTED consumers will never see those messages.
来源：https://stackoverflow.com/questions/45064622/guarantee-unique-global-transaction-for-kafka-producers