Rx operator to distinct sequences

Question

IMPORTANT: for a description of the results and some more details, please have a look also to my answer

I need to group and filter a sequence of objects

Answer 1

I'm not sure if this does exactly what you'd like, but you may be to group the elements explicitly using the group keyword, and then to manipulate the various IObservables separately before recombining them.

E.g. if we have class definitions such as

class A
{
    public char Key { get; set; }
}

class X : A { }
...

and a Subject<A>

Subject<A> subject = new Subject<A>();

then we can write

var buffered =
    from a in subject
    group a by new { Type = a.GetType(), Key = a.Key } into g
    from buffer in g.Buffer(TimeSpan.FromMilliseconds(300))
    where buffer.Any()
    select new
    {
        Count = buffer.Count,
        Type = buffer.First().GetType().Name,
        Key = buffer.First().Key
    };

buffered.Do(Console.WriteLine).Subscribe();

We can test this with the data you provided:

subject.OnNext(new X { Key = 'a' }); 
Thread.Sleep(100);
subject.OnNext(new X { Key = 'b' }); 
Thread.Sleep(100);
subject.OnNext(new X { Key = 'a' }); 
Thread.Sleep(100);
...
subject.OnCompleted();

To get the output you provided:

{ Count = 2, Type = X, Key = a }
{ Count = 1, Type = X, Key = b }
{ Count = 1, Type = Y, Key = b }
{ Count = 1, Type = Y, Key = c }
{ Count = 2, Type = Z, Key = a }
{ Count = 2, Type = Z, Key = c }
{ Count = 1, Type = Z, Key = b }

Answer 2

Not sure if this is exactly what you want, but it seems to support your use cases.

First, let's define the base class to use (you can easily modify this to suit your needs):

public class MyEvent
{
    public string NetworkAddress { set; get; }
    public string EventCode { set; get; }
}

Let's set up your devices as an array of IObservable<MyEvent> - you may have these available differently, and the below would need to change to accommodate that of course. These devices will each produce a value with a random delay between 0.5 and 1.5 seconds.

var deviceA = new MyEvent[] { new MyEvent() {NetworkAddress = "A", EventCode = "1"},
                              new MyEvent() {NetworkAddress = "A", EventCode = "1"},
                              new MyEvent() {NetworkAddress = "A", EventCode = "2"} };

var deviceB = new MyEvent[] { new MyEvent() {NetworkAddress = "B", EventCode = "1"},
                              new MyEvent() {NetworkAddress = "B", EventCode = "2"},
                              new MyEvent() {NetworkAddress = "B", EventCode = "2"},
                              new MyEvent() {NetworkAddress = "B", EventCode = "3"} };   

var random = new Random();                                 

var deviceARand = deviceA.ToObservable().Select(a => Observable.Return(a).Delay(TimeSpan.FromMilliseconds(random.Next(500,1500)))).Concat();
var deviceBRand = deviceB.ToObservable().Select(b => Observable.Return(b).Delay(TimeSpan.FromMilliseconds(random.Next(500,1500)))).Concat();

var devices = new IObservable<MyEvent>[] { deviceARand, deviceBRand };

Now let's take all of these individual device streams, make them 'distinct', and merge them into a single master stream:

var stream = devices.Aggregate(Observable.Empty<MyEvent>(), (acc, device) => acc.DistinctUntilChanged(a => a.EventCode).Merge(device));

Once you have that, getting this stream to be consumed periodically is just a matter of buffering it with Buffer:

stream.Buffer(TimeSpan.FromSeconds(1)).Subscribe(x => { /* code here works on a list of the filtered events per second */ });

Answer 3

After searches and experiments, I put together some code that produces the output that I expect:

static void Main(string[] args)
    {
        const string Address1 = "A:2.1.1";
        const string Address2 = "A:2.1.2";
        var comparer = new EventComparer();
        var eventMessageA1 = new EventMessage { NetworkAddress = Address1, EventCode = 1, Attribute = 4 };
        var eventMessageB1 = new EventMessage { NetworkAddress = Address2, EventCode = 1, Attribute = 5 };
        var eventMessageA2 = new EventMessage { NetworkAddress = Address1, EventCode = 1, Attribute = 5 };
        var list = new[] { eventMessageA1, eventMessageA1, eventMessageB1, eventMessageA2, eventMessageA1, eventMessageA1 };

        var queue = new BlockingCollection<EventMessage>();
        Observable.Interval(TimeSpan.FromSeconds(2)).Subscribe
            (
                l => list.ToList().ForEach(m =>
                {
                    Console.WriteLine("Producing {0} on thread {1}", m, Thread.CurrentThread.ManagedThreadId);
                    queue.Add(m);
                })
            );

        // subscribing
        queue.GetConsumingEnumerable()
            .ToObservable()
             .Buffer(TimeSpan.FromSeconds(5))
             .Subscribe(e =>
                 {
                     Console.WriteLine("Queue contains {0} items", queue.Count);
                     e.Distinct(comparer).ToList().ForEach(m =>
                  Console.WriteLine("{0} - Consuming: {1} (queue contains {2} items)", DateTime.UtcNow, m, queue.Count));
                 }
             );

        Console.WriteLine("Type enter to exit");
        Console.ReadLine();
    }

    public class EventComparer : IEqualityComparer<EventMessage>
    {
        public bool Equals(EventMessage x, EventMessage y)
        {
            var result = x.NetworkAddress == y.NetworkAddress && x.EventCode == y.EventCode && x.Attribute == y.Attribute;
            return result;
        }

        public int GetHashCode(EventMessage obj)
        {
            var s = string.Concat(obj.NetworkAddress + "_" + obj.EventCode + "_" + obj.Attribute);
            return s.GetHashCode();
        }
    }

    public class EventMessage
    {
        public string NetworkAddress { get; set; }

        public byte EventCode { get; set; }

        public byte Attribute { get; set; }

        public override string ToString()
        {
            const string Format = "{0} ({1}, {2})";
            var s = string.Format(Format, this.NetworkAddress, this.EventCode, this.Attribute);
            return s;
        }
    }

Anyway, monitoring the application, it seems that this causes a memory leak. My question is now:

• what is causing the memory leak? [please see the update below]
• is this the best way to do it (if I put the distinct on the first observable, I don't get the other events on next buffers, but items in each buffer should be isolated from others)?
• how can I write a test using the test scheduler?

UPDATE:

it seems that the memory increment lasts only some minutes, then the value is stable. I will run a long test. Of course, this would be an absolutely acceptable behavior.

UPDATE 26.08.12:

• as I already mentioned in the previous update, the memory usage increases only (and slowly) for some minutes after the startup. After 8 hours the memory consumed was stable, with normal fluctuations in the range of few KB)
• this question is very similar to mine and the suggested Drain extension could apply well to my problem (still to be verified)

Anyway, I think that my question is still open for unit tests using the test scheduler.

thanks Francesco