Getting stacktrace of all threads without attaching GDB

Question

Is there a way to print stacktrace of all threads without attaching GDB?

Or is there a command which I can use as gdb batch mode to print stacktrace of all threads?

Answer 1

pstack?

Usage:

pstack <pid>

From the man page:

pstack - print a stack trace of a running process
...
If the process is part of a thread group, then pstack will print out a stack trace for each of the threads in the group.

Answer 2

The following script works for me on linux Please note that it first finds out the directory of the executable and changes to that directory (because executable can have shared libraries with relative path by using -Wl,-rpath,$dir specified during linking, and you would like gdb to find the symbols for these shared libraries for the stack trace). It also assumes that gdb is present on the system.

#!/bin/bash

pid=$1

EXE=`readlink -f /proc/$pid/exe`

DIR=`dirname $EXE`

cd $DIR

gdb $EXE --batch -ex "attach $pid" -ex "thread apply all bt"

Answer 3

As in linux All threads are essentially Light weight processes , they get individual LWP PIDs. you can dump the status of these by using a ps or top based script or echoing the w in /proc/sysrq-trigger. This will give you the I/O state of the threads in the /var/log/messages.

This won't give you the complete stack trace though . but its a good way to debug the thread states on periodical intervals.

you can also explore whats current scheduler state by looking into /proc/sched_debug to see which tasks are getting scheduled and so on .

Answer 4

elfutils among other utilities includes eu-stack:

eu-stack
    Print a stack for each thread in a process or core file.

It is faster than gdb or pstack, which is just a shell script wrapper around gdb. To print stack traces run eu-stack as follows:

$ eu-stack -p 2209
PID 2209 - process
TID 2209:
#0  0x00007f53476b667b __poll
#1  0x00007f5348f98e99 g_main_context_iterate.isra.23
#2  0x00007f5348f99232 g_main_loop_run
#3  0x000055e604b1e56a main
#4  0x00007f53475cc00a __libc_start_main
#5  0x000055e604b1e76a _start
TID 2223:
#0  0x00007f53476b667b __poll
#1  0x00007f5348f98e99 g_main_context_iterate.isra.23
#2  0x00007f5348f98fac g_main_context_iteration
#3  0x00007f5348f98ff1 glib_worker_main
#4  0x00007f5348fc0486 g_thread_proxy
#5  0x00007f534813761b start_thread
#6  0x00007f53476c2c2f __clone
TID 2224:
#0  0x00007f53476b667b __poll
#1  0x00007f5348f98e99 g_main_context_iterate.isra.23
#2  0x00007f5348f99232 g_main_loop_run
#3  0x00007f5349581b56 gdbus_shared_thread_func
#4  0x00007f5348fc0486 g_thread_proxy
#5  0x00007f534813761b start_thread
#6  0x00007f53476c2c2f __clone

Answer 5

I've been using this:

pidof program | xargs -n1 sudo gdb --batch -ex "thread apply all bt" -p

Answer 6

There is a thread apply all command in GDB:

(gdb) thread apply all bt
Thread 12 (Thread 0x7f7fe2116700 (LWP 5466)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
#1  0x0000000000425358 in ?? ()
...
Thread 1 (Thread 0x7f7feabc27c0 (LWP 5465)):
#0  0x00007f7fe76c5203 in select () at ../sysdeps/unix/syscall-template.S:82

Sadly, GDB seems not to be able to read the commands from a pipe, so to run the commands in its batch mode, a temporary file must be used:

$ gdbbt() {
  tmp=$(tempfile)
  echo thread apply all bt >"$tmp"
  gdb -batch -nx -q -x "$tmp" -p "$1"
  rm -f "$tmp"
}
$ gdbbt $(pidof $SHELL)