Only question marks in backtrace reported by gdb on ARM

Question

I'm trying to debug a software with gdbserver on ARM to get a backtrace of a crash. Unfortunately I get only question marks. Everywhere, I read this problem is simply related to the lack of symbols, but symbols are not stripped from my libraries.

If I try to use the file command to load the symbols in the client I get:

reading symbols from <path>/libQtWebKit.so.4.7.2...(no debugging symbols found)...done.

and then, when the crash occurs:

Program received signal SIGSEGV, Segmentation fault.
0x00000000 in ?? ()
(gdb) bt
#0  0x00000000 in ?? ()
#1  0x4bf38b88 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

My libraries are compiled in release but the symbols are actually there. With nm I can find those. Why do I only get question marks? Is this only because the libraries are compiled with optimization? Isn't it possible to debug with libraries in release mode?

Answer 1

The corrupt stack note is probably your problem. It looks like a return address or virtual table entry or something was overwritten with zeros, and then control was transferred there. Even if you have symbols available, those addresses aren't pointing to valid symbols. Hence the segfault.

I don't envy your task. These are some of the hardest bugs to track down, and can even move or temporarily go away when you make code changes to try and catch them. Your best bet is usually something like git bisect or your VCS equivalent to find the commit that introduced it. Hopefully it isn't too difficult to reproduce.

Answer 2

One trick you can sometimes use when you get the "SEGV at address 0" problem is to manually pop the return address from the top of the stack into the pc and trying to do a stack trace from there. This assumes that you got to address 0 by doing an indirect call through a NULL pointer, which is the most common way of getting to address 0.

Now I'm not too familiar with ARM, but on an x86 PC, you would do:

(gdb) set $eip = *(void **)$esp
(gdb) set $esp = $esp + 4

and then do another backtrace to figure out where you really are.

If you can figure out the calling convention used for ARM by your compiler, you should be able to do something similar.