How scalable are Google Club Pub/Sub compared Object Change Notifications

Question

As the title asks, how scalable is Google Club Pub/Sub compared to Object Change Notifications in Google Cloud Storage when using Signed URLs to upload objects?

How do e

Answer 1

At 1000 object changes per second, you want to use Cloud Pub/Sub notifications.

Both Object Change Notifications and Cloud Pub/Sub notifications will work just fine at 1000 QPS. However, at those sorts of load, there are a few reasons to prefer Cloud Pub/Sub.

First, Cloud Pub/Sub subscriptions support pulling messages. With one call to pull(), you can retrieve 100 or more messages at once, and then acknowledge them all with one call to acknowledge(). Object Change Notifications always make one call to your service per message. Using Cloud Pub/Sub can instantly reduce the number of RPCs your server needs to deal with by two orders of magnitude.

Second, at high QPS, you will want to start considering failures, timeouts, and retries. Cloud Pub/Sub is a superior option here as well, for a variety of reasons. For one, it supports configurable ack deadlines, while object change notifications always must be processed within 20 seconds. For another, you can query to see the size of your current backlog, in case you get behind. If you're using push subscriptions, Cloud Pub/Sub also has a much friendlier traffic ramp-up if your message receiver goes offline for a bit, so you don't overwhelm your own servers as they come back online.

Third, flexibility. Cloud Pub/Sub is designed around this use case. Receiving a high volume of messages is their whole bread and butter, and there are many features and libraries focused on it. Cloud Storage, though, is focused on storing data. Object change notifications works, but it will never offer as many notification features as Cloud Pub/Sub.