spark streaming checkpoint recovery is very very slow

Question

• Goal: Read from Kinesis and store data in to S3 in Parquet format via spark streaming.
• Situation: Application runs fine initially, running batches of 1hour and th

Answer 1

I had similar issues before, my application getting slower and slower.

try to release memory after using rdd, call rdd.unpersist() https://spark.apache.org/docs/latest/api/java/org/apache/spark/rdd/RDD.html#unpersist(boolean)

or spark.streaming.backpressure.enabled to true

http://spark.apache.org/docs/latest/streaming-programming-guide.html#setting-the-right-batch-interval

http://spark.apache.org/docs/latest/streaming-programming-guide.html#requirements

also, check your locality setting, maybe too much data move around.