node and Error: EMFILE, too many open files

Question

For some days I have searched for a working solution to an error

Error: EMFILE, too many open files

It seems that many people have the same proble

Answer 1

For nodemon users: Just use the --ignore flag to solve the problem.

Example:

nodemon app.js --ignore node_modules/ --ignore data/

Answer 2

I had this issue, and i solved it by running npm update and it worked.

In some cases you may need to remove node_modules rm -rf node_modules/

Answer 3

For anyone that might still be looking for solutions, using async-await worked fine for me:

fs.readdir(<directory path></directory>, async (err, filenames) => {
    if (err) {
        console.log(err);
    }

    try {
        for (let filename of filenames) {
            const fileContent = await new Promise((resolve, reject) => {
                fs.readFile(<dirctory path + filename>, 'utf-8', (err, content) => {
                    if (err) {
                        reject(err);
                    }
                    resolve(content);
                });
            });
            ... // do things with fileContent
        }
    } catch (err) {
        console.log(err);
    }
});

Answer 4

Like all of us, you are another victim of asynchronous I/O. With asynchronous calls, if you loop around a lot of files, Node.js will start to open a file descriptor for each file to read and then will wait for action until you close it.

File descriptor remains open until resource is available on your server to read it. Even if your files are small and reading or updating is fast, it takes some time, but in the same time your loop don't stop to open new files descriptor. So if you have too many files, the limit will be soon reached and you get a beautiful EMFILE.

There is one solution, creating a queue to avoid this effect.

Thanks to people who wrote Async, there is a very useful function for that. There is a method called Async.queue, you create a new queue with a limit and then add filenames to the queue.

Note: If you have to open many files, it would be a good idea to store which files are currently open and don't reopen them infinitely.

const fs = require('fs')
const async = require("async")

var q = async.queue(function(task, callback) {
    console.log(task.filename);
    fs.readFile(task.filename,"utf-8",function (err, data_read) {
            callback(err,task.filename,data_read);
        }
    );
}, 4);

var files = [1,2,3,4,5,6,7,8,9,10]

for (var file in files) {
    q.push({filename:file+".txt"}, function (err,filename,res) {
        console.log(filename + " read");
    });
}

You can see that each file is added to the queue (console.log filename), but only when the current queue is under the limit you set previously.

async.queue get information about availability of the queue through a callback, this callback is called only when data file is read and any action you have to do is achieved. (see fileRead method)

So you cannot be overwhelmed by files descriptor.

> node ./queue.js
0.txt
    1.txt
2.txt
0.txt read
3.txt
3.txt read
4.txt
2.txt read
5.txt
4.txt read
6.txt
5.txt read
7.txt
    1.txt read (biggest file than other)
8.txt
6.txt read
9.txt
7.txt read
8.txt read
9.txt read

Answer 5

Building on @blak3r's answer, here's a bit of shorthand I use in case it helps other diagnose:

If you're trying to debug a Node.js script that is running out of file descriptors here's a line to give you the output of lsof used by the node process in question:

openFiles = child_process.execSync(`lsof -p ${process.pid}`);

This will synchronously run lsof filtered by the current running Node.js process and return the results via buffer.

Then use console.log(openFiles.toString()) to convert the buffer to a string and log the results.

Answer 6

Had the same problem when running the nodemon command so i reduced the name of files open in sublime text and the error dissappeared.