for loop over event driven code?

Question

In a redis datastore I have a list of keys, I want to iterate over that list of keys and get those values from redis. The catch is I am using an event driven language, javascript via node.js

If javascript were procedural I could do this

function getAll(callback) {
    var list = redis.lrange(lrange('mykey', 0, -1);
    for ( var i = 0; i < list.length; i+= 1 ) {
        list[i] = redis.hgetall(list[i]);
    }
    callback(list);
}

But, I cannot, therefor.. I do this?

function getAll(callback) {
    redis.lrange('mykey', 0, -1, function(err, reply) {
        // convert reply into messages
        var list = [];
        var index = -1;
        var recurse = function() {
            if ( index == reply.length ) {
                callback(list);
            } else {
                redis.hgetall(reply[i], function(err, reply) {
                    list.push(reply);
                    index += 1;
                    recurse();
                });
            }
        };
        recurse()
    });
};

This seams wrong, because, instead of executing all requests at once, and then letting callbacks insert onto list, I am forcing a sequential call sequence. What happens if there are 1000s of keys?

Could I do this some how?

function getAll(callback) {
    redis.lrange('mykey', 0, -1, function(err, reply) {

        // convert reply into messages
        var list = [];
        var insert = function(err, reply) {
            list.push(reply);
        };
        for ( var i = 0; i < reply.length; i += 1 ) {
            redis.hgetall(reply[i], insert);
        }

        ??? how to block until finished ??? 
        callback(list);
    });
};

Answer 1

Declare an object variable before you dispatch your calls within the for loop. Each call can than add its result into the object.

You then need code to wait for all calls to be done. This might help you: https://gist.github.com/464179

Example:

function getAll(callback) {

    var results = [];

    var b = new Barrier(2, function() {
        // all complete callback
        callback();
        }, function() {
        // Aborted callback, not used here
    });

    var list = redis.lrange(lrange('mykey', 0, -1);
    for ( var i = 0; i < list.length; i+= 1 ) {
        //dispatch your call
        call(function(foo){
            results.push(foo);
            b.submit();
        });
    }
}

Please note that call() should be your async database function, that executes the callback on result.

Answer 2

??? how to block until finished ???

You can't, not if the calls you're making are asynchronous. You have to define your getAll with the expectation it will complete asynchronously, then recast it a bit.

I'm not familiar with the redis calls you're making, but the fundamental pattern is:

function doTheBigThing(param, callbackWhenDone) {
    asyncCallToGetList(param, function(result) {
        var list = [];

        asyncCallToGetNextEntry(result.thingy, receivedNextEntry);

        function receivedNextEntry(nextResult) {
            if (nextResult.meansWeAreDone) {
                callback(list);
            }
            else {
                list.push(nextResult.info);
                asyncCallToGetNextEntry(result.thingy, receivedNextEntry);
            }
        }
    });
}

Breaking that down:

1. doBigThing (your getAll) is called.
2. It does the "get the list" call, passing in the function to use as the callback when we have the list or list handle or whatever.
3. That callback defines a function, receivedNextEntry, and calls the "get the next entry" function with whatever info is used to retrieve the entry, passing that in as a callback.
4. receivedNextEntry stores the entry it got and, if done, fires the main "all done" callback; if not, it issues the next request.

Sorry not to be able to give you a redis-specific answer, but I think the mappings are:

• doBigThing = getAll
• asyncCallToGetList = redis.lrange
• asyncCallToGetNextEntry = redis.hgetall

...but what the parameters are you use with redis.lrange and redis.hgetall I'm afraid I don't know.

Answer 3

If you find yourself needing to use patterns like this often, then you may be interested in trying out the async.js library. Using async.js you could write something like this:

function getAll(callback) {
    redis.lrange('mykey', 0, -1, function(err, reply) {
        async.concat(reply, redis.hgetall, callback);
    });
};

Which basically means "call hgetall on each item in 'reply' then concat all the results and pass to the callback".