How to queue http get requests in Nodejs in order to control their rate?

Question

I have a NodeJS app which sends HTTP get requests from various places in the code, some are even dependent (sending a request, waiting for a reply, processing it and based on re

Answer 1

I would use Deferreds and return one for every queued request. You can then add succeed/fail callbacks to the deferred promise after it has been queued.

var deferred = queue.add('http://example.com/something');
deferred.fail(function(error) { /* handle failure */ });
deferred.done(function(response) { /* handle response */ });

You can hold a [ url, deferred ] pairs in your queue, and each time you dequeue a URL you'll also have the Deferred that goes with it, which you can resolve or fail after you process the request.