How to separate initial data load from incremental children with Firebase?

Question

I have an application where new children get added to Firebase every 5 seconds or so. I have thousands of children.

On application load, I\'d like to process the ini

Answer 1

Since child_added events for the initial, pre-loaded data will fire before the value event fires on the parent, you can use the value event as a sort of "initial data loaded" notification. Here is some code I slightly modified from another similar StackOverflow question.

var initialDataLoaded = false;
var ref = new Firebase('https://<your-Firebase>.firebaseio.com');

ref.on('child_added', function(snapshot) {
  if (initialDataLoaded) {
    var msg = snapshot.val().msg;
    // do something here
  } else {
    // we are ignoring this child since it is pre-existing data
  }
});

ref.once('value', function(snapshot) {
  initialDataLoaded = true;
});

Thankfully, Firebase will smartly cache this data, meaning that creating both a child_added and a value listener will only download the data one time. Adding new Firebase listeners for data which has already crossed the wire is extremely cheap and you should feel comfortable doing things like that regularly.

If you are worried about downloading all that initial data when you don't actually need it, I would follow @FrankvanPuffelen's suggestions in the comments to use a timestamped query. This works really well and is optimized using Firebase queries.

Answer 2

Added a createdAt timestamp in the database and limited the child_added with the current time and it seems working fine to me.

const onChildAdded = database()
  .ref(refURL)
  .limitToLast(constValues.LAST_MESSAGE_ADDED)
  .orderByChild('createdAt')
  .startAt(date.getTime())
  .on('child_added', (snapshot) => {
    parseMessage(snapshot);
  });

database()
  .ref(refURL)
  .limitToLast(constValues.LAST_MESSAGES_LIMIT)
  .once('value', (snapshot) => {
    parseInitialMessages(snapshot);
    setLoading(false);
  });

Answer 3

There are two possible solutions to this problem, neither satisfying:

1) Timestamp your children and only request children that have a timestamp greater than a "now" value. This is not great because you may have a synchronicity issue between your application's "now" and the "now" on whatever is pushing the data to Firebase or the Firebase server value for "now".

2) Use value and child added. Any duplicates seen in child_added that had already been seen in value may be discarded. This is unuseable for large data sets as it requires downloading all historical data TWICE!

Why can't there be a command like "child_added" that DOESN'T give everything ever?

Answer 4

Improved the answer from @jacobawenger to not use a global state variable

var ref = new Firebase('https://<your-Firebase>.firebaseio.com');

ref.once('value', function(snapshot) {
  // do something here with initial value

  ref.on('child_added', function(snapshot) {
    // do something here with added childs
  });

});