Execute a computing-intensive task in background

Question

I have a task that requires a lot of computing without IO while executing and I don\'t want to block thread while executing this task. How can I do this in Node.js

Answer 1

The usual way to remove compute intensive stuff from blocking the main thread in node.js is to pass that work off to another process. There are many ways to structure that.

1. Clustering where you have a group of processes all designed to execute the same thing and each one gets a different incoming request.
2. Create a specific child process designed specifically to execute a given task and create that child process that runs and then completes for the duration of the task whenever this particular task needs to be done.
3. Create a group of child processes and a queue of work and have each child process grab a work item from the queue, execute it, communicate back the result, then grab the next item from the queue, etc...

And, these techniques can be combined and one in combination too.

In the first case, you are just creating more processes all doing the same thing.

In the other two cases, you are specifically moving the compute intensive work to child processes so the main process is not burdened or blocked by that work.

There are a number of libraries that help provide an interface for using child processes. One such library provides the webWorker type of interface.

Answer 2

You need background workers and some kind of communication protocol between your web instance and worker instances. For example you can use kue to create tasks queue

// web worker
var kue = require('kue');
var queue = kue.createQueue();

....

app.post('/calculateSomeCrazyStuff', (req, res, err) => {
  queue.create('myLongRunningJobName1', {
    fibonacciSequence: 52
  }).save((err) => {
    if (err) {
      return next(err);
    }
    res.status(204).send();
  });
});



// jobs worker
var kue = require('kue');
var queue = kue.createQueue();


queue.process('myLongRunningJobName1', function(job, done) {
  // calculate fibonacci
  done();
});