问题
I have a test application in .NET that creates kafka producer and sends a few messages to a topic using Confluent client library.
For some reason the acknowledgement for the first message always arrives 1 second late, acknowledgements for subsequent messages arrive almost immediately.
Is this a normal behavior or am I missing some configuration?
- I think I tried tuning all producer configs, nothing helps except setting EnableDeliveryReports = false.
- Topic is not partitioned and already exists at the moment of sending the first message.
If I update the application to send messages to a few topics, then first message to each topic gets sent with 1 second delay.
static void Main() { var producer = new Producer<Null, string>(new ProducerConfig { BootstrapServers = "localhost:9092", LingerMs = 100, BatchNumMessages = 1, }); for (var i = 0; i < 10; i++) { var start = DateTime.Now; producer.ProduceAsync( new TopicPartition("test-topic", Partition.Any), new Message<Null, string> { Value = $"hello kafka! #{i}" }).Wait(2000); var now = DateTime.Now; Console.WriteLine($"{now:HH:mm:ss.fff} Message sent in {(now - start).TotalMilliseconds:N1}."); } producer.Dispose(); Console.ReadLine(); }
Sample output:
18:06:13.605 Message sent in 1,007.0.
18:06:13.607 Message sent in 1.0.
18:06:13.608 Message sent in 1.0.
18:06:13.609 Message sent in 1.0.
18:06:13.610 Message sent in 1.0.
18:06:13.611 Message sent in 1.0.
18:06:13.612 Message sent in 1.0.
18:06:13.613 Message sent in 1.0.
18:06:13.614 Message sent in 1.0.
18:06:13.615 Message sent in 1.0.
回答1:
After some more digging in kafka request logs I found that before sending the first message producer sends API_VERSIONS and METADATA requests for all topics. After 1 second it sends again METADATA request, this time for the specific topic that producer is trying to push message to.
Setting TopicMetadataRefreshIntervalMs to something less than 1 second reduces the first message delay. As a downside producer starts sending metadata requests more often. Surprisingly setting TopicMetadataRefreshIntervalMs to something more than a second does not extend the delay.
Not sure if this is a bug or a feature.
来源:https://stackoverflow.com/questions/53728258/kafka-producer-is-slow-on-first-message