问题
There is a way to enable graceful shutdown of spark streaming by setting property spark.streaming.stopGracefullyOnShutdown to true and then kill the process with kill -SIGTERM command. However I don't see such option available for structured streaming (SQLContext.scala).
Is the shutdown process different in structured streaming? Or is it simply not implemented yet?
回答1:
This feature is not implemented yet. But the write ahead logs of spark structured steaming claims to recover state and offsets without any issues.
回答2:
This Feature is not implemented yet and also it will give you duplicates if you kill the job from resource Manager while the batch is running.
来源：https://stackoverflow.com/questions/48567281/shutdown-spark-structured-streaming-gracefully