Kafka Consumer Marking the coordinator 2147483647 dead

Question

I am using Kafka Server 0.9 with consumer kafka-client version 0.9 and kafka-producer 0.8.2.

Every thing is working great except i am getting lot of info that the coordi

Answer 1

I faced this issue today and solved it (temporarily, might I add). I've posted an answer here on how I did it.

Answer 2

In my case the message was in logs when I try to assign partitions manually. After I've read in api docs of the new consumer follow notice:

It is also possible for the consumer to manually assign specific partitions (similar to the older "simple" consumer) using assign(Collection). In this case, dynamic partition assignment and consumer group coordination will be disabled.

That is, if you have code like this:

    KafkaConsumer<String, String> consumer = new KafkaConsumer(props);
    consumer.assign( Arrays.asList(
            new TopicPartition("topic", 0),
            new TopicPartition("topic", 1)
    ));

then the message "Marking the coordinator 2147483647 dead" puts in our logs always.

Answer 3

I have faced the same issue. Finally after follow Shannon recommendation about TRACING logs, I used:

logging.level.org.apache.kafka=TRACE

To find out that my client was trying to resolve Euler:9092 as coordinator... Local name!! So I commented out and changed listeners and advertised.listeners values in server.properties file. It is working now! :-)

Answer 4

Marking the coordinator dead happens when there is a Network communication error between the Consumer Client and the Coordinator (Also this can happen when the Coordinator dies and the group needs to rebalance). There are a variety of situations (offset commit request, fetch offset, etc) that can cause this issue. I will suggest that you research what's causing this situations

Answer 5

This is basically you are not able to reach to Kafka.
In my case I was running Kafka in vagrant box, and if I start VPN it refresh
vagrant ip hence it was not able to connect to it.
Possible Solution: In this case stop VPN and start your vagrant.

Answer 6

This may also be related to a long garbage collection stop-the-world phase. In my case I encountered this message after > 10 sec GCs.