Spark streaming not remembering previous state

Question

I wrote spark streaming program with stateful transformation. It seems like my spark streaming application is doing computation correctly with check pointing. But if i terminate

Answer 1

As it is described in the checkpointing documentation you have to adjust your code to be able to restore state from the checkpoints.

In particular you cannot create StreamingContext directly but have to use StreamingContext.getOrCreate method which takes:

• checkpoint directory
• function which can be used to set up the context (Unit => StreamingContext)