Spark Structured Stream get messages from only one partition of Kafka

Question

I got the situation when spark can stream and get messages from only one partition of Kafka 2-patition topic.

My topics: C:\\bigdata\\kafka_2.11-0.10.1.1\\bin\\win

Answer 1

There is a known Kafka issue in 0.10.1.* client: https://issues.apache.org/jira/browse/KAFKA-4547

Right now you can use 0.10.0.1 client as a workaround. It can talk to a Kafka 0.10.1.* cluster.

See https://issues.apache.org/jira/browse/SPARK-18779 for more details.