NodeJS http.request not returning data even after specifying return on the 'end' event

Question

Basically I am trying to scrap some data from website and perform the DOM extraction, deletion and updation on a callback function binded to the \'end\' event of http.

Answer 1

The simplest modification that will make this work is to pass a callback function to extractEmail to receive the data once it's ready.

var scraper = {
    extractEmail: function (directoryName, cb) {
        var result = getDirectory(directoryName);
        if (result !== 404) {
            var protocol = result.https ? https : http;
            protocol.request({
                host: 'somevalue.net',
                method: "GET"
            }, function (res) {
                var data = '';
                res.on('data', function (chunk) {
                    data += chunk;
                });

                res.on('end', function () {
                    cb(null, data);
                });
            })
                .on('error', function (err) {
                    cb(err);
                })
                .end();
        }
        else {
            cb(new Error('Failed'));
        }
    }
};

And use it like this:

app.get('/:directory', function (req, res) {
    scraper.extractEmail(req.params.directory, function(err, n) {
        if (err) return console.error(err);
        console.log(n);
        res.send(n);
    });
});

Answer 2

You cannot return a value from an asynchronous callback. Well, you can, but it most likely will get ignored and most certainly won't do what you want.

You cannot even return a promise in that place. You can only resolve a promise where you now use the return statements. You need to return a promise from the main function and then resolve or reject the promise in your event handlers where you use returns now.

For more info see those answers:

• Return Promise result instead of Promise in Nodejs
• Return value in function from a promise block
• jQuery: Return data after ajax call success

Answer 3

Is your 'var scraper' also in the route.js file? I guess it's not and you are unable to access that other js file, for doing so use module.exports.

eg.

// module.js
var name = "foobar";
// export it
exports.name = name; 



Then, in route.js...
>      //route.js
>      // get a reference to your required module
>      var myModule = require('./module'); 
>      //correct path to folder where your above file is
>      // name is a member of myModule due to the export above
>      var name = myModule.name;