Per request logging in Node.js

Question

I am an experienced Java developer picking up Node.js and making the shift to the asynchronous model. Most things are going fine except for logging. I cannot find anything sim

Answer 1

I got into this same problem some time ago, and finally I could spend sometime researching it. The @ibash approach and his post put me in the lead to solve the problem I had (thanks for your help). I only walked some steps more in order to print in the logs automatically a unique id per request. In your case you can add origin and destination IP and all information needed to each request, using same approach and print it automatically in all logs.

My approach: - As @ibash explained, I used continuation-local-storage to share information among all the modules per request. So I generate a unique id per request and store it in a namespace created with this library - I wrapped the Winston library (in a very simple way) in order to recover the information from the namespace shared and override all Winston methods I use adding to the string the unique Id. Obviously in your case you should add all the info you need and you have stored previously in the namespace of the library.

As the problem was a little complex to explain to people no familiarize with all these things, I wrote it down in a post with a clear example that you can reuse if you want. Winston wrap could be really useful: Express.js: Logging info with global unique request ID – Node.js

I hope you can reuse my code and perhaps in the future Express implements a solution for this.

Answer 2

Answering this as I just wrote a post on how to use continuation-local-storage to save a "transaction id" with every log (without manually propagating it). You can do the same for the client ip, process id, etc.

Follow this post: https://datahero.com/blog/2014/05/22/node-js-preserving-data-across-async-callbacks/

But instead of just saving a transaction id, you'll want these as well: request.connection.remoteAddress and process.pid

Let me know if you have any questions here or there, and I'll answer them.

Answer 3

These instructions were from an Ubuntu 12.04 distribution I set up, but they should apply pretty closely to RHEL, Fedora, CentOS, etc.

Rsyslog is a system logging utility you can use to log messages from any program on a Linux machine. First you need to find your rsylog configuration information. You can do that with the following command:

sudo find / -name rsyslog.conf

If you can't find the configuration file, you can list the service running to see if rsyslog is even on your machine with the following command:

service --status-all

Now open the file it finds and do the following:

• Comment out the line $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
• Uncomment $ModLoad imtcp
• Uncomment $InputTCPServerRun and specify port number 1514, going to use 1514 b/c Ubuntu 12.04 rsyslog has a problem dropping permissions if I use port 514, other distributions don't have similar issues and you could keep default port #. I get around this by using iptables to reroute port 514 traffic to 1514
• Change $FileCreateMode 0640 to 0644

Now I created a file named /etc/rsyslog.d/10.conf (this is a secondary configuration file for rsyslog where we can filter message, name log files, etc) and added the following to it:

$template DailyPerHostLogs,"/var/log/MyLogFile_%$YEAR%_%$MONTH%_%$DAY%.log"

#:msg,contains,"MsgName" -?DailyPerHostLogs
*.* -?DailyPerHostLogs
&~

This file creates a new file for each day and finds any message sent with MsgName in the text and puts it into the daily file and then removes it from the queue to be logged by any other log requests so we don't double log it.

Now you can reboot the machine you are working on and it all should be working. You can check this by looking for files in /var/log as defined in 10.conf above. Hit the logger from the command line by issuing the following commands:

logger this is from the command line
echo "this is from the tcp port" > /dev/tcp/127.0.0.1/1514

You should see both those lines pop up in the log file. If you get that, then let's move on to the node module that will be able to hit the log.

var net = require('net');

var client = net.connect({port: this.1514}, function(){ console.log("Open"); });

client.write(' ' + "sMsgName: What"+ ' ' + "hath" + ' ' + "God wrought?" + '\n');

//Do everything else your program needs. . . 

The '\n' on the write tells rsyslog we are done with this line. Also, you will need to prepend a space for the filtering to work: http://www.rsyslog.com/log-normalization-and-the-leading-space/

The devil is always in the details with a setup like this, but I think this will get you most of the way there and google searching will get you the rest of the way.