Firestore DeadlineExceeded exception for big collections

Question

I\'m trying to read bigger collections from Google Firestore for testing and archiving purposes. I\'m hitting some interesting errors when I try to get all documents from co

Answer 1

After some help from the firebase support team we were able to figure out that there is indeed a bug with the python client api. There is a bugfix coming in one of the next releases. Most likely it will enable the python library to sort by documentid and therefore use start_after().

Up until then you have two possible solutions:

1. use another field to sort on and use start_after()

2. use the node.js library with paging like:

var db = admin.firestore();
admin.firestore().settings({ timestampsInSnapshots: true });
function readNextPage(lastReadDoc) {
  let query = db
    .collection(collection)
    .orderBy(admin.firestore.FieldPath.documentId())
    .limit(100);
}

Answer 2

In my case I got this error just getting the entire collection. It was not even that big of a collection but I guess the documents in the collection are large. I did a paginated update. This was a node firebase function:

let lastReadDoc = false;


let lastDoc: string = '';
  const limitRecordCount = 10;
  do {
    await db
      .collection('something/' + somethingId + '/somethingcollection')
      .orderBy('id')
      .limit(limitRecordCount)
      .startAfter(lastDoc)
      .get()
      .then((snapshot: any) => {
        let counter = 0;
        snapshot.docs.forEach((doc: any) => {
          const docData = doc.data();
          if (lastDoc !== docData.id) {
            lastDoc = docData.id;
            counter = counter + 1;
          }
          // ***********************
          // business logic per doc here
          // ***********************
        });
        if (counter < limitRecordCount) {
          lastReadDoc = true;
        }
      })
      .catch((err: any) => {
        lastReadDoc = true;
        console.log('Error getting booking documents', err);
      });
  } while (lastReadDoc === false);