ERROR Error when sending message to topic

Question

While producing message in kafka, i am getting the following error :

$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic nil_PF1_P1
hi
hello         


        

Answer 1

I get the same error today with confluent_kafka 0.9.2 (0x90200) and librdkafka 0.9.2 (0x90401). In my case, I specified the wrong broker port in tutorialpoints example:

$ kafka-console-producer.sh --broker-list localhost:9092 --topic tutorialpoint-basic-ops-01

although my broker was started on port 9094:

$ cat server-02.properties 
broker.id=2
port=9094
log.dirs=/tmp/kafka-example-logs-02
zookeeper.connect=localhost:2181

Although the 9092 port was not open (netstat -tunap), it took 60s for kafka-console-producer.sh to raise an error. Looks like this tool needs a fix to:

• fail faster
• with a more explicit error message.

Answer 2

I use Apache Kafka on a Hortonworks (HDP 2.X release) installation. The error message encountered means that Kafka producer was not able to push the data to the segment log file. From a command-line console, that would mean 2 things :

1. You are using incorrect port for the brokers
2. Your listener config in server.properties are not working

If you encounter the error message while writing via scala api, additionally check connection to kafka cluster using telnet <cluster-host> <broker-port>

NOTE: If you are using scala api to create topic, it takes sometime for the brokers to know about the newly created topic. So, immediately after topic creation, the producers might fail with the error Failed to update metadata after 60000 ms.

I did the following checks in order to resolve this issue:

The first difference once I check via Ambari is that Kafka brokers listen on port 6667 on HDP 2.x (apache kafka uses 9092).

listeners=PLAINTEXT://localhost:6667

Next, use the ip instead of localhost. I executed netstat -na | grep 6667

tcp        0      0 192.30.1.5:6667        0.0.0.0:*               LISTEN     
tcp        1      0 192.30.1.5:52242       192.30.1.5:6667        CLOSE_WAIT 
tcp        0      0 192.30.1.5:54454       192.30.1.5:6667        TIME_WAIT

So, I modified the producer call to user the IP and not localhost:

./kafka-console-producer.sh --broker-list 192.30.1.5:6667 --topic rdl_test_2

To monitor if you have new records being written, monitor the /kafka-logs folder.

cd /kafka-logs/<topic name>/
ls -lart
-rw-r--r--.  1 kafka hadoop        0 Feb 10 07:24 00000000000000000000.log
-rw-r--r--.  1 kafka hadoop 10485756 Feb 10 07:24 00000000000000000000.timeindex
-rw-r--r--.  1 kafka hadoop 10485760 Feb 10 07:24 00000000000000000000.index

Once, the producer successfully writes, the segment log-file 00000000000000000000.log will grow in size.

See the size below:

-rw-r--r--. 1 kafka hadoop 10485760 Feb 10 07:24 00000000000000000000.index
-rw-r--r--. 1 kafka hadoop       **45** Feb 10 09:16 00000000000000000000.log
-rw-r--r--. 1 kafka hadoop 10485756 Feb 10 07:24 00000000000000000000.timeindex

At this point, you can run the consumer-console.sh:

./kafka-console-consumer.sh --bootstrap-server 192.30.1.5:6667 --topic rdl_test_2 --from-beginning
response is hello world

After this step, if you want to produce messages via the Scala API's , then change the listeners value(from localhost to a public IP) and restart Kafka brokers via Ambari:

listeners=PLAINTEXT://192.30.1.5:6667 

A Sample producer will be as follows:

package com.scalakafka.sample
import java.util.Properties
import java.util.concurrent.TimeUnit

import org.apache.kafka.clients.producer.{ProducerRecord, KafkaProducer}
import org.apache.kafka.common.serialization.{StringSerializer, StringDeserializer}

class SampleKafkaProducer {
  case class KafkaProducerConfigs(brokerList: String = "192.30.1.5:6667") {
    val properties = new Properties()
    val batchsize :java.lang.Integer = 1

    properties.put("bootstrap.servers", brokerList)
    properties.put("key.serializer", classOf[StringSerializer])
    properties.put("value.serializer", classOf[StringSerializer])
    //    properties.put("serializer.class", classOf[StringDeserializer])
        properties.put("batch.size", batchsize)
    //    properties.put("linger.ms", 1)
    //    properties.put("buffer.memory", 33554432)
  }

  val producer = new KafkaProducer[String, String](KafkaProducerConfigs().properties)

  def produce(topic: String, messages: Iterable[String]): Unit = {
    messages.foreach { m =>
      println(s"Sending $topic and message is $m")
      val result = producer.send(new ProducerRecord(topic, m)).get()
      println(s"the write status is ${result}")
    }
    producer.flush()
    producer.close(10L, TimeUnit.MILLISECONDS)
  }
}

Hope this helps someone.

Answer 3

adding such a line after the topic helped with the same issue: ... --topic XXX --property "parse.key = true" --property "key.separator =:"

Hope this helps someone.

Answer 4

I faced similar problem, where I was able to produce and consume on localhost but not from different machines on network. Based few answers I got the clue that essentially we need to expose advertised.listener to producer and consumer, however giving 0.0.0.0 was also not working. So gave exact IP against advertised.listeners

advertised.listeners=PLAINTEXT://HOST.IP:9092

And I left listener=PLAINTEXT://:9092 as it is.

So with this the spark exposes advertised ip and port to producers and consumers

Answer 5

If you are running hortonworks cluster, check the listening port in ambari.

In my case 9092 was not my port. I went to ambari and found the listening port was set to 6667 it worked for me. :)

Answer 6

I know this is old but this may work for someone else who's dealing with it: I changed 2 things:
1. change the "bootstrap.servers" property or the --broker-list option to 0.0.0.0:9092
2. change (uncomment and edit in my case) the server.properties in 2 properties

• listeners = PLAINTEXT://your.host.name:9092 to listeners=PLAINTEXT://:9092
• advertised.listeners=PLAINTEXT://your.host.name:9092 to advertised.listeners=PLAINTEXT://localhost:9092