buffer while processing items

Question

I have an event that fires regularly. Let\'s assume that processing the event takes ~1s. Instead of waiting 1s for each received event I want to accumulate events until

Answer 1

This is non-trival. Fortunately @DaveSexton has already done all the hard work. You want BufferIntrospective from the Rxx library. Check out the source here.

The reason why this is hard is because IObserver doesn't have built-in means to signal back-pressure - other than the subtlety of the blocking of OnXXX invocations. The Observable needs to pay attention to the Observer, and you need to introduce concurrency to manage the buffering.

Also note that if you have multiple subscribers, they will get different data as what they receive depends on both the source event rate and their consumption rate.

Another approach is to just add all the events to a thread-safe queue in your OnNext handler, and have a separate task that empties the queue in a loop. BufferIntrospective is probably cleaner though.

Had a little play, and this toy implementation seems to work. But Rxx will be more robust, so this is just pedagogical really to show what sort of thing is involved. The key is the introduction of concurrency via the scheduler.

public static IObservable> BufferIntrospective(
    this IObservable source,
    IScheduler scheduler = null)
{
    scheduler = scheduler ?? Scheduler.Default;
    return Observable.Create>(o => {
        Subject feedback = new Subject();
        var sourcePub = source.Publish().RefCount();
        var sub = sourcePub.Buffer(
            () => feedback).ObserveOn(scheduler).Subscribe(@event =>
            {                
                o.OnNext(@event);
                feedback.OnNext(Unit.Default);
            },
            o.OnError,
            o.OnCompleted);
        var start = sourcePub.Take(1).Subscribe(_ => feedback.OnNext(Unit.Default));
        return new CompositeDisposable(sub, start);
    });        
}

This sample code shows the usage and how two differently paced subscribers get different buffering of events, one receiving batches of 5, the other batches of 10.

I am using LINQPad's Dump to show the contents of each buffer easily.

var xs = Observable.Interval(TimeSpan.FromSeconds(0.2)).Take(30);

var buffered = xs.BufferIntrospective();

buffered.Subscribe(x => {
    x.Dump();
    Task.Delay(TimeSpan.FromSeconds(1)).Wait();
});

buffered.Subscribe(x => {
    x.Dump();
    Task.Delay(TimeSpan.FromSeconds(2)).Wait();
});