RxJS, how to poll an API to continuously check for updated records using a dynamic timestamp

Question

I am new to RxJS and I am trying to write an app that will accomplish the following things:

1. On load, make an AJAX request (faked as fetchItems() f

Answer 1

You seem to mean that modifiedSince is part of the state you carry, so it should appear in the scan. Why don-t you move the action in do into the scan too?. Your seed would then be {modifiedSince: null, itemArray: []}.

Errr, I just thought that this might not work, as you need to feed modifiedSince back to the fetchItem function which is upstream. Don't you have a cycle here? That means you would have to use a subject to break that cycle. Alternatively you can try to keep modifiedSince encapsulated in a closure. Something like

function pollItems (fetchItems, polling_frequency) {
var modifiedSince = null;
var data$ = Rx.Observable.interval(polling_frequency)
  .startWith('run right away')
  .flatMap(function() { 
    // `fetchItems(modifiedSince)` returns an array of items modified after `modifiedSince`
    return fetchItems(modifiedSince);
  })
  .do(function(item) {
    if(item.updatedAt > modifiedSince) {
      modifiedSince = item.updatedAt;
    }
  })
  .scan(function(previous, current) {
    previous.push(current);
    return previous;
  }, []);

return data$;
}

I have to run out to celebrate the new year, if that does not work, I can give another try later (maybe using the expand operator, the other version of scan).

Answer 2

How about this:

var interval = 1000;
function fetchItems() {
    return items;
}

var data$ = Rx.Observable.interval(interval)
  .map(function() { return fetchItems(); })
  .filter(function(x) {return x.lastModified > Date.now() - interval}
  .skip(1)
  .startWith(fetchItems());

That should filter the source only for new items, plus start you off with the full collection. Just write the filter function to be appropriate for your data source.

Or by passing an argument to fetchItems:

var interval = 1000;
function fetchItems(modifiedSince) {
    var retVal = modifiedSince ? items.filter( function(x) {return x.lastModified > modifiedSince}) : items
    return retVal;
}

var data$ = Rx.Observable.interval(interval)
  .map(function() { return fetchItems(Date.now() - interval); })
  .skip(1)
  .startWith(fetchItems());

Answer 3

Here is another solution which does not use closure or 'external state'.

I made the following hypothesis :

• fetchItems returns a Rx.Observable of items, i.e. not an array of items

It makes use of the expand operator which allows to emit values which follow a recursive relationship of the type x_n+1 = f(x_n). You pass x_n+1 by returning an observable which emits that value, for instance Rx.Observable.return(x_n+1) and you can finish the recursion by returning Rx.Observable.empty(). Here it seems that you don't have an ending condition so this will run forever.

scan also allows to emit values following a recursive relationship (x_n+1 = f(x_n, y_n)). The difference is that scan forces you to use a syncronous function (so x_n+1 is synchronized with y_n), while with expand you can use an asynchronous function in the form of an observable.

Code is not tested, so keep me updated if this works or not.

Relevant documentation : expand, combineLatest

var modifiedSinceInitValue = // put your date here
var polling_frequency = // put your value here
var initial_state = {modifiedSince: modifiedSinceInitValue, itemArray : []}
function max(property) {
  return function (acc, current) {
    acc = current[property] > acc ? current[property] : acc;
  }
}    
var data$ = Rx.Observable.return(initial_state)
  .expand (function(state){
             return fetchItem(state.modifiedSince)
                   .toArray()
                   .combineLatest(Rx.Observable.interval(polling_frequency).take(1), 
                     function (itemArray, _) {
                       return {
                         modifiedSince : itemArray.reduce(max('updatedAt'), modifiedSinceInitValue), 
                         itemArray : itemArray
                       }
                     }

  })