Firestore pagination - how to find if there are more data after query (using limit)

Question

We use ndb datastore in our current python 2.7 standard environment. We migrating this application to python 3.7 standard environment with firestore (native mode).

We us

Answer 1

There is no direct equivalent in Firestore pagination. What you can do instead is fetch one more document than the N documents that the page requires, then use the presence of the N+1 document to determine if there is "more". You would omit the N+1 document from the displayed page, then start the next page at that N+1 document.

Answer 2

I build a custom firestore API not long ago to fetch records with pagination. You can take a look at the repository. This is the story of the learning cycle I went through:

My first attempt was to use limit and offset, this seemed to work like a charm, but then I walked into the issue that it ended up being very costly to fetch like 200.000 records. Because when using offset, google charges you also for the reads on all the records before that. The Google Firestore Pricing Page clearly states this:

There are no additional costs for using cursors, page tokens, and limits. In fact, these features can help you save money by reading only the documents that you actually need.

However, when you send a query that includes an offset, you are charged a read for each skipped document. For example, if your query uses an offset of 10, and the query returns 1 document, you are charged for 11 reads. Because of this additional cost, you should use cursors instead of offsets whenever possible.

My second attempt was using a cursor to minimize those reads. I ended up fetching N+1 documents and place the cursor like so:

collection = 'my-collection'
cursor = 'we3adoipjcjweoijfec93r04' # N+1th doc id
q = db.collection(collection)
snapshot = db.collection(collection).document(cursor).get()
q = q.start_at(snapshot) # Place cursor at this document
docs = q.stream()

But this required a little more code and I thought it could be done more easily. So why not fetch N documents, get the document ID of the last document and using the start_after function?

collection = 'my-collection'
cursor = 'dkfjweoidkdfjowepp39jfe' # Nth doc id
q = db.collection(collection)
snapshot = db.collection(collection).document(cursor).get()
q = q.start_after(snapshot) # Place cursor after this document
docs = q.stream()