Apache Spark’s Structured Streaming with Google PubSub

问题

I'm using Spark Dstream to pull and process data from Google PubSub.

I'm looking for a way to move to structured streaming but still using Pub/Sub.

Also, I should mention that my messages are Snappy compressed in Pub/Sub.

I found this issue which claims that using Pub/Sub with structured streaming is not supported.

Is someone has encountered this problem? Is it possible to implement custom Receiver to read the data from Pub/Sub

Thanks

回答1:

The feature request you referenced is still accurate: Cloud Pub/Sub does not have the concept of an offset to track your read position, so structured streaming with Cloud Pub/Sub is not supported.

来源：https://stackoverflow.com/questions/55964723/apache-spark-s-structured-streaming-with-google-pubsub