Kafka producer is slow on first message

孤人 提交于 2021-01-27 11:44:46

问题


I have a test application in .NET that creates kafka producer and sends a few messages to a topic using Confluent client library.

For some reason the acknowledgement for the first message always arrives 1 second late, acknowledgements for subsequent messages arrive almost immediately.

Is this a normal behavior or am I missing some configuration?

  • I think I tried tuning all producer configs, nothing helps except setting EnableDeliveryReports = false.
  • Topic is not partitioned and already exists at the moment of sending the first message.
  • If I update the application to send messages to a few topics, then first message to each topic gets sent with 1 second delay.

    static void Main()
    {
        var producer = new Producer<Null, string>(new ProducerConfig
        {
            BootstrapServers = "localhost:9092",
            LingerMs = 100,
            BatchNumMessages = 1,
        });
    
        for (var i = 0; i < 10; i++)
        {
            var start = DateTime.Now;
            producer.ProduceAsync(
                new TopicPartition("test-topic", Partition.Any), new Message<Null, string>
                {
                    Value = $"hello kafka! #{i}"
                }).Wait(2000);
    
            var now = DateTime.Now;
            Console.WriteLine($"{now:HH:mm:ss.fff} Message sent in {(now - start).TotalMilliseconds:N1}.");
        }
    
        producer.Dispose();
        Console.ReadLine();
    }
    

Sample output:

18:06:13.605 Message sent in 1,007.0.
18:06:13.607 Message sent in 1.0.
18:06:13.608 Message sent in 1.0.
18:06:13.609 Message sent in 1.0.
18:06:13.610 Message sent in 1.0.
18:06:13.611 Message sent in 1.0.
18:06:13.612 Message sent in 1.0.
18:06:13.613 Message sent in 1.0.
18:06:13.614 Message sent in 1.0.
18:06:13.615 Message sent in 1.0.

回答1:


After some more digging in kafka request logs I found that before sending the first message producer sends API_VERSIONS and METADATA requests for all topics. After 1 second it sends again METADATA request, this time for the specific topic that producer is trying to push message to.

Setting TopicMetadataRefreshIntervalMs to something less than 1 second reduces the first message delay. As a downside producer starts sending metadata requests more often. Surprisingly setting TopicMetadataRefreshIntervalMs to something more than a second does not extend the delay.

Not sure if this is a bug or a feature.



来源:https://stackoverflow.com/questions/53728258/kafka-producer-is-slow-on-first-message

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!