The best pattern for handling async looping in Node.js

Question

I\'m new to Node and trying to ensure that I\'m using sane designs for a JSON-driven web app.

I\'ve got a bunch of data stored in Redis and am retrieving it through

Answer 1

but I can't help thinking that the i == replies.length-1 is a little untidy?

I've heard a lot of people say that. This is how I would do it by hand:

app.get("/facility", function(req, res, next) {
  rc.keys("FACILITY*", function(err, replies) {
    if (err) return next(err);
    var pending = replies.length;
    res.write("[");
    replies.forEach(function (reply) {
      rc.get(reply, function(err, reply) {
        res.write(reply);
        if (!--pending) {
          res.write("]");
          return res.end();
        }
        res.write(",");
      });
    });
  });
});

Obviously doing it by hand isn't too pretty, which is why people have it abstracted into a library or some other function. But like it or not, that is how you do an async parallel loop. :)

You can use the async library mentioned before to hide the nasty innards.

Answer 2

You can use the async library, it provides some handy methods for looping, such as forEach:

forEach(arr, iterator, callback)

Applies an iterator function to each item in an array, in parallel. The iterator is called with an item from the list and a callback for when it has finished. If the iterator passes an error to this callback, the main callback for the forEach function is immediately called with the error.

Note, that since this function applies the iterator to each item in parallel there is no guarantee that the iterator functions will complete in order.

Example

// assuming openFiles is an array of file names and saveFile is a function
// to save the modified contents of that file:

async.forEach(openFiles, saveFile, function(err){
    // if any of the saves produced an error, err would equal that error
});

Answer 3

The above code might not do what you expect. You're kicking off each .get() in sequence, but they might not call back in sequence — so the results could stream out in any order. If you want to stream the results instead of collecting them in memory, you need to .get() in sequence.

I think that caolan’s async library makes a lot of this easier. Here’s one way you could use it to get each item in sequence (warning, untested):

app.get("/facility", function(req, res) {
    rc.keys("FACILITY*", function(err, replies) {
        var i = 0;
        res.write("[");
        async.forEachSeries(replies, function(reply, callback){
            rc.get(reply, function(err, reply) {
                if (err){
                    callback(err);
                    return;
                }
                res.write(reply);
                if (i < replies.length) {
                    res.write(",");
                }
                i++;
                callback();
            });
        }, function(err){
            if (err) {
                // Handle an error
            } else {
                res.end(']');
            }
        });
    });
});

If you don’t care about the order, just use async.forEach() instead.

If you wouldn’t mind collecting the results and want them to return in sequence, you could use async.map() like this (warning, also untested):

app.get("/facility", function(req, res) {
    rc.keys("FACILITY*", function(err, replies) {
        async.map(replies, rc.get.bind(rc), function(err, replies){
            if (err) {
                // Handle an error
            } else {
                res.end('[' + replies.join(',') + ']');
            }
        });
    });
});