How to filter dstream using transform operation and external RDD?

Question

I used transform method in a similar use case as described in Transform Operation section of Transformations on DStreams:

spamI         


        

Answer 1

The difference between the Spark doc example and your code is the use of ssc.checkpoint().

Although the specific code example you provided will work without checkpoint, I guess you actually require it. But the concept of introducing an external RDD into the scope of a checkpointed DStream is potentially invalid: when recovering from a checkpoint, the external RDD may have changed.

I tried to checkpoint the external RDD, but I had no luck with it either.