Node.js/Express and parallel queues

Question

We are building an infrastructure which features a Node.js server and Express.

In the server, what is happening is as follow:

1. The server accepts an inc

Answer 1

(Answering my own question)

According to this question on Stack Overflow a solution in my case would be to implement a queue using Caolan McMahon's async module.

The main application will create jobs and push them into a queue, which has a limit on the number of concurrent jobs that can run. This allows processing tasks concurrently but with a strict control on the limit. It works like Cocoa's NSOperationQueue on Mac OSX.

Answer 2

You can use Kue module with Redis(database to hold the jobs) Backing the queue. you create jobs and place them in a using kue module and you can put how many ever workers to work on them. useful links : kue - https://github.com/Automattic/kue

Answer 3

To do this, i would use a structure like the one Heroku provides with Web/Worker Dynos (servers). The web servers can accept the requests and pass the info on to the workers, who can do the information processing and uploading. I would have the front-end site listen on a socket (socket.io) for the url of the external CDN which will be fired from the worker when the upload is finished. Hopefully that makes sense.

Answer 4

tldr; You can use the native Node.js cluster module to handle a lot of concurrent requests.

Some preamble: Node.js per se is single threaded. Its Event Loop is what makes it excellent for handling multiple requests simultaneosly even in its single thread model is, which is one of its best features IMO.

The real deal: So, how can we scale this to even handle more concurrent conections and use all CPUs available? With the cluster module.

This module will work exactly as pointed by @Qualcuno, which will allows you to create multiple workers (aka process) behind the master to share the load and use more efficiently the CPUs availables.

According with Node.js official documentation:

Because workers are all separate processes, they can be killed or re-spawned depending on your program's needs, without affecting other workers. As long as there are some workers still alive, the server will continue to accept connections.

The required example:

var cluster = require('cluster');
var http = require('http');
var numCPUs = require('os').cpus().length;

if (cluster.isMaster) {
  // Fork workers.
  for (var i = 0; i < numCPUs; i++) {
    cluster.fork();
  }

  cluster.on('exit', function(worker, code, signal) {
    console.log('worker ' + worker.process.pid + ' died');
  });
} else {
  // Workers can share any TCP connection
  // In this case its a HTTP server
  http.createServer(function(req, res) {
    res.writeHead(200);
    res.end("hello world\n");
  }).listen(8000);
}

Hope this is what you need.

Comment if you have any further questions.