Avoiding callback hell with multiple streams

Question

How can I avoid using a recursion like structure when I got several streams to open and I have to get an absolute end event to finish the logic.

Answer 1

Use a counter:

var someArray = ['file1', 'file2', 'file3'];
var still_processing = someArray.length;

someArray.forEach(function( file ) {
    fs.createReadStream( file )
        .pipe( /* do some stuff */ )
        .on('end', function() {
            still_processing--;

            if (!still_processing) {
                // done
            }
        });
}

This is the basic mechanism. This control flow pattern is encapsulated by the async.parallel() function in async.js:

var someArray = ['file1', 'file2', 'file3'];
var streams_to_process = [];

someArray.forEach(function( file ) {
    streams_to_process.push(function(callback) {
        var result = "";
        fs.createReadStream( file )
            .pipe( /* do some stuff */ )
            .on('end', function() {
                callback(null, result);
            });
    });
});

async.parallel(streams_to_process, function(err, results) {
    // all done
});

Internally, async.parallel uses a counter captured in a closure to keep track of when all async processes (in this case the 'end' events) are done.

There are other libraries for this. Most promise libraries for example provide an .all() method that works the same way - internally keeping track of a counter value and firing the .then() callback when all is done.

Answer 2

I feel like there is a way to do this with promises but I don't know how.

Yes, there is. As promises do represent asynchronous values, you'd get a promise for the end of one stream:

var p = new Promise(function(resolve, reject) {
    fs.createReadStream(file)
    .on('error', reject)
    .pipe(/* do some stuff */)
    .on('end', resolve)
    .on('error', reject); // call reject(err) when something goes wrong
});

It could be used like p.then(functio(usageInfo) { console.log("stream ended"); }).

Now if you create multiple promises, one for filename in your array, all the streams will run in parallel and resolve their respective promise when done. You then can use Promise.all to collect - read "await" - all the results from each of them into a new promise for an array of results.

var promises = ['file1', 'file2', 'file3'].map(function(file) {
    return new Promise(function(resolve, reject) {
        …
    });
});
Promise.all(promises).then(function(usageInfos) {
    console.log("all of them done", usageInfos),
}, function(err) {
    console.error("(At least) one of them failed", err);
});