Tools to analyze core dump from Node.js

Question

If I use gcore to make a code dump of a Node.js process, what are the best tools to analyze it?

Inspired by: Tool for analyzing java core dump

In my specific cas

Answer 1

For investigating crashes, I've found node-segfault-handler to be invaluable. It's a module I cooked up to get a native-code stack trace in the event of a hard-crash with a signal - eg deref of NULL leading to SIGSEGV

For investigating memory / allocation issues, here's some of the data I've collected thus far:

1) Blog post by Dave Patheco - the author talks about using a plugin to MDB for getting JS stacks and such. Sadly, as far as I can tell, the source of that plugin was never released (nor any binary form).

2) Postmortem Debugging in Dynamic Environments - ACM Queue article also written by Dave Patheco (linked from the blog post). While it makes for GREAT background reading, the article doesn't have many concrete tools and techniques in it.

3) node-panic - A pure-JS tool for dumping state in the event of an assert-failure type crash. Does nothing to help debug crashes that originate from native code faults (SIGSEGV, etc)

4) Joyent: Debugging Production Systems - talk by Bryan Cantrill on the tools and techniques he recommends (thx crickeys).

Answer 2

On Linux and Mac you can use llnode a plugin for the lldb debugger. The project is available under the nodejs organization on github:

https://github.com/nodejs/llnode

You can install from source via github or use brew on Mac. The readme on github should help you get it installed and there's an introductory blog article here:

https://developer.ibm.com/node/2016/08/15/exploring-node-js-core-dumps-using-the-llnode-plugin-for-lldb/

The original question was about memory analysis and the v8 findjsobjects and v8 findjsinstances commands will help there by generating a basic histogram of object counts and allowing you to list the instances of each type.

There's a full article on using llnode for memory analysis here: http://www.brendangregg.com/blog/2016-07-13/llnode-nodejs-memory-leak-analysis.html

Answer 3

2017 update: Now you can use @h-hellyer's solution (llnode, based on lldb rather than mdb). https://stackoverflow.com/a/40045103/3221630

mdb + mdb_v8 is the way to go.

In order to use mdb, you will need a supported OS.

Now, most likely you will be running on Linux. If this is your case:

Part 1. get your core dump

You can get your core dump in many ways. To get your core dump from a running process you can do this:

pgrep -lf node # get pids
gdb -p your_pid

# once in gdb..
gcore  # this will output your core dump
detach # this will allow the process to continue to run.

Part 2. use mdb

There is a chance you know about Solaris, OpenSolaris, IllumOS or SmartOS. Most likely this is not the case. If you can afford the time of setting up SmartOS and mdb_v8, fine.

If not, install VirtualBox, and then autopsy. This handles the ritual of installing SmartOS as well as uploading your core dump files to the VM.

Once you are done, and when you are in your mdb session, you can then follow some of the steps from this presentation.