spark streaming checkpoint recovery is very very slow

Question

• Goal: Read from Kinesis and store data in to S3 in Parquet format via spark streaming.
• Situation: Application runs fine initially, running batches of 1hour and th

Answer 1

raised a Jira issue : https://issues.apache.org/jira/browse/SPARK-19304

The issue is because we read more data per iteration than what is required and then discard the data. This can be avoided by adding a limit to getResults aws call.

Fix: https://github.com/apache/spark/pull/16842