TransactionId prefix for producer-only and read-process-write - ProducerFencedException

问题

Background: We have been getting ProducerFencedException in our producer-only transactions, and want to introduce uniqueness to our prefix to prevent this issue.

In this discussion, Gary mentions that in the case of read-process-write, the prefix must be the same in all instances and after each restart. How to choose Kafka transaction id for several applications, hosted in Kubernetes?

While digging into this issue, I came to the realisation that we are sharing the same prefixId for both producer-only and read-process-write.

In our TopicPublisher class wrapping kafkaTemplate, we already have a publish() and publishInTransaction() methods for read-process-write and producer-only use cases respectively.

I am thinking to have 2 sets of kafkaTemplates/TransactionManagers/ProducerFactories, one with a fixed prefixId to be used by the publish() method and one with a unique prefix to be used in publishInTransaction().

My question is:

Does the prefix for producer-only need to be the same after a pod is restarted. Can we just append some uuid or k8s podId? Someone mentioned there may be delays with aborting transactions.
Is there a clean way to detect if the TopicPublisher is being called from a KafkaListener, so we can have just 1 publish method that uses the correct kafkaTemplate as needed?

回答1:

Actually, there is no issue using the same transactionIdPrefix, at least with recent versions.

The factory gets a txIdPrefix.

For read-process-write, we create (and cache) a producer with transactionalId:

    private String zombieFenceTxIdSuffix(String topic, int partition) {
        return this.consumerGroupId + "." + topic + "." + partition;
    }

which is suffixed onto the prefix.

For producer only-transactions, we create (and cache) a producer with the prefix and simple numeric suffix.

In the upcoming 2.3 release, there is also an option to assign a producer to a thread so the same thread always uses the same transactional.id.

I believe it needs to be the same, unless you don't mind waiting for transaction.timeout.ms (default 1 minute).

The maximum amount of time in ms that the transaction coordinator will wait for a transaction status update from the producer before proactively aborting the ongoing transaction.If this value is larger than the transaction.max.timeout.ms setting in the broker, the request will fail with a InvalidTransactionTimeout error.

This is what we do in spring-integration-kafka


if (this.transactional
        && TransactionSynchronizationManager.getResource(this.kafkaTemplate.getProducerFactory()) == null) {
    sendFuture = this.kafkaTemplate.executeInTransaction(t -> {
        return t.send(producerRecord);
    });
}
else {
    sendFuture = this.kafkaTemplate.send(producerRecord);
}

You can also use String suffix = TransactionSupport.getTransactionIdSuffix(); which is what the factory uses when it is asked for producer - if null, you are not running on a transactional consumer thread.

来源：https://stackoverflow.com/questions/57504481/transactionid-prefix-for-producer-only-and-read-process-write-producerfencedex

标签

spring-kafka