Spark 2.1 Structured Streaming - Using Kakfa as source with Python (pyspark)

Question

With Apache Spark version 2.1, I would like to use Kafka (0.10.0.2.5) as source for Structured Streaming with pyspark:

kafka_app.py:

from pyspark.sql imp         


        

Answer 1

You need to use sql-structured streaming jar "spark-sql-kafka-0-10_2.11-2.1.0.jar" instead of spark-streaming-kafka-0-10-assembly_2.10-2.1.0.jar.