Google Pub/Sub Subscriber not receiving messages after a while

Question

I have a simple python script that uses Google pubsub to detect new files in the google cloud storage. The script simply adds new messages to a queue where another thread proces

Answer 1

Apart from the flow control suggestion I offered in my previous comment, you could also define a Cloud Function that gets triggered any time a new message is published in a Pub/Sub topic. These Cloud Functions act as subscriptions and will get notified every time a certain event (such as a message being published) occurs.

This tutorial will help you to develop a background Cloud Function that will get triggered when a message is published in a Pub/Sub topic.

Answer 2

In general, there can be several reasons why a subscriber may stop receiving messages:

1. If a subscriber does not ack or nack messages, the flow control limits can be reached, meaning no more messages can be delivered. This does not seem to be the case in your particular instance given that you immediately ack messages. As an aside, I would recommend against acking messages before your queue has processed them unless you are okay with the possibility of messages not being processed. Otherwise, if your app crashes after the ack, but before the message queue processes them, you will have not processed the message and will not get it redelivered since it was acked.
2. If another subscriber starts up for the same subscription, it could be receiving the messages. In this scenario, one would expect the subscriber to receive a subset of the messages rather than no messages at all.
3. Publishers just stop publishing messages and therefore there are no messages to receive. If you restart the subscriber and it starts receiving messages again, this probably isn't the case. You can also verify that a backlog is being built up by looking at the Stackdriver metric for subscription/backlog_bytes.

If your problem does not fall into one of those categories, it would be best to reach out to Google Cloud support with your project name, topic name, and subscription name so that they can narrow down the issue to either your user code, the client library, or the service.

Answer 3

I got stuck 1 hour on this problem, so this is how I fixed my problem :

The GOOGLE_APPLICATION_CREDENTIALS environment variable was setup to a different service account, who wasn't on the right project

project_id = "my_project_sandbox" 

And

my_project.json (service account used by project)

{
  "type": "service_account",
  "project_id": "my_project_prod",
  "private_key_id": "---",
  "private_key": "---",
  ...
}