Add new field or change the structure on all Firestore documents

Question

Consider a collection of users. Each document in the collection has name and email as fields.

{
  \"users\": {
    \"uid1\         


        

Answer 1

You fell into a gap of NOSQL databases: Document oriented databases do not guarantee structural integrity of the data (as RDBMS do)

The deal is:

• in an RDBMS all stored data have the same structure at any given time (within the same instance or cluster). When changing the structure (ER-diagram) you have to migrate the data for all existing records which costs time and effort.

  In result you application can be optimised for the current version of the data structure.

• in a Document oriented database each record is an independent "Page" with its own independent structure. If you change the structure it only applies to new documents. So you don't need to migrate the existing data.

  In result you application must be able to deal with all versions of your data structure you ever used in your current database.

I don't know about firebase in detail but in general you never update a document in a NOSQL database. You only create a new version of the document. So even if you update all documents your application must be prepared to deal with the "old" data structure...

Answer 2

To solve this, you need to update each user to have the new property and for that I recommend you to use a Map. If you are using a model class when you are creating the users as explained in my answer from this post, to update all users, just iterate over the users collection amd use the following code:

Map<String, Object> map = new HashMap<>();
map.put("timestamp", FieldValue.serverTimestamp());
userDocumentReference.set(map, SetOptions.merge());

Answer 3

I am guessing that last_login is a primitive data type, maybe a long to hold a timestamp. An auto-generated setter would look like this:

private long last_login;

public void setLast_login(long last_login) {
    this.last_login = last_login;
}

This leads to a crash when old documents that lack the field are fetched due to null assignment to a variable of a primitive data type. One way around it is to modify your setter to pass in a variable of the equivalent wrapper class - Long instead of long in this case, and put a null check in the setter.

private long last_login;

public void setLast_login(Long last_login) {
    if(last_login != null) {
        this.last_login = last_login;
    }
}

The cost of avoiding the null pointer exception is the boxing-unboxing overhead.

Answer 4

Just wanted to share because I read that you were hoping for a Firestore based solution.

This worked for me. forEach will query each document in the collection and you can manipulate as you like.

    db.collection("collectionName").get().then(function(querySnapshot) {
        querySnapshot.forEach(async function(doc) {
            await db.collection("collectionName").doc(doc.id).set({newField: value}, {merge: true});
            // doc.data() is never undefined for query doc snapshots
            console.log(doc.id, " => ", doc.data());
        });
    });

Answer 5

I wrote some routines to help automate this process back when I posted the question. I did not post them since these are a bit rudimentary and I was hoping for an elegant Firestore-based solution. Because such solution is not still available, here are the functions I wrote.

In short, we have functions for renaming a field, adding a field, or deleting a field. To rename a field, different functions are used depending on the data type. Maybe someone could generalise this better? The functions below are:

• add_field: Adds a field in all documents of a collection.
• delete_field: Deletes a field in all documents of a collection.
• rename_*_field: Renames a field containing a certain data type (*) in all documents of a collection. Here I include examples for String, Integer, and Date.

Add field:

public void add_field (final String key, final Object value, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {
                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            Map<String, Object> new_map = new HashMap<>();
                            new_map.put(key, value);
                            batch.update(docRef, new_map);
                        }
                        batch.commit();
                    } else {
                        // ... "Error adding field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting documents -> " + e
                }
            });
}

Delete field:

public void delete_field (final String key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            Map<String, Object> delete_field = new HashMap<>();
                            delete_field.put(key, FieldValue.delete());
                            batch.update(docRef, delete_field);
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices -> " + e
                }
            });
}

Rename field:

public void rename_string_field (final String old_key, final String new_key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            String old_value = document.getString(old_key);

                            if (old_value != null) {
                                Map<String, Object> new_map = new HashMap<>();
                                new_map.put(new_key, old_value);

                                Map<String, Object> delete_old = new HashMap<>();
                                delete_old.put(old_key, FieldValue.delete());

                                batch.update(docRef, new_map);
                                batch.update(docRef, delete_old);
                            }
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices ->" + e
                }
            });
}

public void rename_integer_field (final String old_key, final String new_key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            int old_value = document.getDouble(old_key).intValue();
                            Integer ov = old_value;
                            if (ov != null) {
                                Map<String, Object> new_map = new HashMap<>();
                                new_map.put(new_key, old_value);

                                Map<String, Object> delete_old = new HashMap<>();
                                delete_old.put(old_key, FieldValue.delete());

                                batch.update(docRef, new_map);
                                batch.update(docRef, delete_old);
                            }
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices -> " + e
                }
            });
}

public void rename_date_field (final String old_key, final String new_key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            Date old_value = document.getDate(old_key);
                            if (old_value != null) {
                                Map<String, Object> new_map = new HashMap<>();
                                new_map.put(new_key, old_value);

                                Map<String, Object> delete_old = new HashMap<>();
                                delete_old.put(old_key, FieldValue.delete());

                                batch.update(docRef, new_map);
                                batch.update(docRef, delete_old);
                            }
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices -> " + e
                }
            });
}