How to do `tail -f logfile.txt`-like processing in node.js?

Question

tail -f logfile.txt outputs the last 10 lines of logfile.txt, and then continues to output appended data as the file grows.

What\'s the recommended way

Answer 1

The canonical way to do this is with fs.watchFile.

Alternatively, you could just use the node-tail module, which uses fs.watchFile internally and has already done the work for you. Here is an example of using it straight from the documentation:

Tail = require('tail').Tail;

tail = new Tail("fileToTail");

tail.on("line", function(data) {
  console.log(data);
});

Answer 2

https://github.com/jandre/always-tail seems a great option if you have to worry about log rotating, example from the readme:

var Tail = require('always-tail');
var fs = require('fs');
var filename = "/tmp/testlog";

if (!fs.existsSync(filename)) fs.writeFileSync(filename, "");

var tail = new Tail(filename, '\n');

tail.on('line', function(data) {
  console.log("got line:", data);
});


tail.on('error', function(data) {
  console.log("error:", data);
});

tail.watch();

Answer 3

you can try to use fs.read instead of ReadStream

var fs = require('fs')

var buf = new Buffer(16);
buf.fill(0);
function read(fd)
{
    fs.read(fd, buf, 0, buf.length, null, function(err, bytesRead, buf1) {
        console.log(buf1.toString());
        if (bytesRead != 0) {
            read(fd);
        } else {
            setTimeout(function() {
                read(fd);
            }, 1000);
        }
    }); 
}

fs.open('logfile', 'r', function(err, fd) {
    read(fd);
});

Note that read calls callback even if there is no data and it just reached end of file. Without timeout you'll get 100% cpu here. You could try to use fs.watchFile to get new data immediately.

Answer 4

Substack has a file slice module that behaves exactly like tail -f, slice-file can stream updates after the initial slice of 10 lines.

var sf = require('slice-file');

var xs = sf('/var/log/mylogfile.txt');
xs.follow(-10).pipe(process.stdout);

Source: https://github.com/substack/slice-file#follow

Answer 5

node.js APi documentation on fs.watchFile states:

Stability: 2 - Unstable. Use fs.watch instead, if available.

Funny though that it says almost the exact same thing for fs.watch:

Stability: 2 - Unstable. Not available on all platforms.

In any case, I went ahead and did yet another small webapp, TailGate, that will tail your files using the fs.watch variant.

Feel free to check it out here: TailGate on github.