java.lang.IllegalStateException: Error reading delta file, spark structured streaming with kafka

Question

I am using Structured Streaming + Kafka for realtime data analytics in our project. I am using Spark 2.2, kafka 0.10.2.

I am facing an issue during streaming query recov

Answer 1

What's your checkpoint location? This is usually because you are using the local file system to store checkpoints. Make sure you set the "checkpointLocation" option and it points to a distributed file system (such as HDFS) that can be accessed by all nodes. [1]

[1] http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing