Kafka Producer - By default supports Multithreading?

Question

I am newbie to kafka. I have created sample kafka sync producer and consumergroup programs using kafka_2.9.2-0.8.1.1. So My question is, do I need to add multithreading code

Answer 1

There are two types of producers available with Kafka. (1) SyncProducer (2) AsyncProducer. If you set the producer.type configuration as async it will uses the AsyncProducers. By default it uses the Synchronous producer class.

Once running in async mode it creates a separate AsyncProducer instance per broker.And each of these AsyncProducer instances maintains its own internal background thread for sending the messages. These are called ProducerSendThread.

So there is one thread running per broker and your parallelism is based on the number of brokers available in the cluster. So adding new brokers in the cluster should provide you the flexibilities to increase the level of parallelism while producing data using Kafka.But remember adding a new broker to your cluster should be considered taking other paramaters also into consideration.