How to change Spark Streaming application with checkpointing?

Question

Please consider the following scenario:

• created initial pipeline via Spark streaming
• enable checkpointing
• run the application for a while

Answer 1

tl;dr Checkpointing is for recovery situations not for upgrades.

From the official documentation about Checkpointing:

A streaming application must operate 24/7 and hence must be resilient to failures unrelated to the application logic (e.g., system failures, JVM crashes, etc.). For this to be possible, Spark Streaming needs to checkpoint enough information to a fault- tolerant storage system such that it can recover from failures.

So to answer your question about using checkpointing (that is meant for fault tolerance) and changing your application code, you should not expect it would work since it is against the design.