Iterate over async function

Question

I have a function that uses the module cherio to get data from a website.

Now I\'d like to iterate this function over an array of keywords, collect the intermediate

Answer 1

Promise is the best solution for handling asynchronous operations.

Promise.all(search_words.map(function(keyword) {
  return new Promise(function(resolve, reject) {
    request(base_url + keyword + "&l=", function(err, resp, body) {
      if (err) {
        return reject(err);
      }
      $ = cheerio.load(body);
      resolve([keyword, $("#searchCount")[0].children[0].data.split(" ").reverse()[0]]);
    });
  });
})).then(function(stats) {
  console.log(stats);
});

Answer 2

Other the usual (and correct) way you use to solve the problem, there are some modules that let you write code that is synchronous, if you really want.

Try to google for "nodejs synchronous" give as result some link to nodejs modules and/or methodologies to write synchronous code in nodejs, but I suppose they are usefull only to some specific problem (never used them myself)

Answer 3

The most common way I can think of is using a promise library like Q.

npm install --save q

Then use it in your code:

var Q = require('q');
var requestFn = q.denodeify(request);

Then you iterate over your values:

var promises = search_words.map(function(keyword) {
   url = base_url + keyword + "&l=";
   return requestFn(url);
});

Q.all(promises).then(function(values) {
    //values will contain the returned values from all requests (in array form)
}, function(rejects) {
   //rejected promises (with errors, for example) land here
});

The denodeify function from Q basically turns the callback-based function into one that returns a promise (a step-in for the future value, as soon as it's there). That function is requestFn (find a better name for it!). All these promises are collected in one Array which is passed to Q.all to make sure that all promises are fulfilled (if one is rejected, other promises are rejected too).

If that is not your intended behavior: There are loads of ways to play with the excellent Q library. See the documentation: https://github.com/kriskowal/q

I did not bullet proof test this code. You might need to play around with it a bit, but it should give you a good idea of how to soundly do things like this. Having a counter go up is usually a very unreliable way of handling asynchronous code.