YDN-DB - Incorrect results using mixed data types with SortedMerge

Question

I'm using the excellent YDN-DB as part of a complicated 'dashboard' web page that allows a user to download a large amount of information and then search & filter the data how they wish.

The data comes down in the form of 1 UNID and 6 columns of visible data which I am storing using a YDN schema like this:

var schema = {
    stores: [
        {
            name: 'test', keyPath: 'unid', autoIncrement: false,
            indexes: [
                {keyPath: 'a', type: 'TEXT'},
                {keyPath: 'b', type: 'TEXT'},
                {keyPath: 'c', type: 'TEXT'},
                {keyPath: 'd', type: 'DATE'},
                {keyPath: 'e', type: 'TEXT'},
                {keyPath: 'f', type: 'TEXT'}
            ]
        }
    ]
};

The user then can then place a filter on any of the six columns above. The code to build the filters looks like this, and uses the YDN KeyRange to filter the data by the values requested by the user.

var filterArr = []
var resultArr = [];

filterArr.push(new ydn.db.IndexIterator('test', 'a', ydn.db.KeyRange.only('search value 1')));
filterArr.push(new ydn.db.IndexIterator('test', 'b', ydn.db.KeyRange.only('search value 2')));
.. [Continue adding filters for more columns if necessary etc] ..

var req = db.scan(new ydn.db.algo.SortedMerge(resultArr), filterArr);

req.then(function() {
    db.values('test', resultArr).done(function(results) {
        .. [Display code goes here] ..
    })
})

So for matched values, this works fine. My problem however, is that the user can specify a date range for column D, so I have changed one of the filterArr lines above like this:

filterArr.push(new ydn.db.IndexIterator('test', 'd', ydn.db.KeyRange.bound(1389571200000, 1390435200000, false, false)));

This leads to incorrect results from the search. It appears to happen only when multiple filterArr values are specified. If I filter column D on its own, it seems to work.

Sorry for the complicated query and really hoping for some kind of answer. I have spent some time debugging the SortedMerge JS and my limited knowledge suggests that something is going wrong with the result of each call to the ydn.db.cmp function, so I wondered whether this was a bug in YDN DB?

Answer 1

From what I can see from you code, column 'd' should be INTEGER, or query should use date like:

ydn.db.KeyRange.bound(toDate(1389571200000),toDate(1390435200000), false, false)

EDIT:

Sorry, key joining algorithm only work for equi-join. Range will not work. You can use one range query however, but require six compound index [a, d], [b, d], [c, d], d, [e, d], [f, d] and join using ZigZeg merge. Detail here http://dev.yathit.com/ydn-db/nosql-query.html You can also find requirement for equi join.

Current ZigZeg is faster then SortedMerge since IndexedDB API spec 1 does not have require methods. However v2 is coming with require methods, namely openKeyCursor and continuePrimaryKey. Result of ZigZeg is sorted by d, whereas SortedMerge is sorted by primary key.

Answer 2

Instead of storing a Date object, store the timestamp representation instead:

var timestamp = new Date().getTime();

Since a timestamp is an integer, you can then effectively use a key range on it.