Google Cloud Pubsub Data lost

Question

I\'m experiencing a problem with GCP pubsub where a small percentage of data was lost when publishing thousands of messages in couple seconds.

I\'m logging both

Answer 1

Talked with some guy from Google, and it seems to be an issue with the Python Client:

The consensus on our side is that there is a thread-safety problem in the current python client. The client library is being rewritten almost from scratch as we speak, so I don't want to pursue any fixes in the current version. We expect the new version to become available by end of June.

Running the current code with thread_safe: false in app.yaml or better yet just instantiating the client in every call should is the work around -- the solution you found.

For detailed solution, please see the Update in the question