Displaying each assembly instruction executed in gdb

Question

I currently have a tricky bug that occurs in a place where I don\'t have access to source or symbols, i.e. I can see the instruction and its address where the crash occurs,

Answer 1

The following should do what you asked for:

# not strictly required, but you'll likely want the log anyway
(gdb) set logging on

# ask gdb to not stop every screen-full
(gdb) set height 0

(gdb) while 1
 > x/i $pc
 > stepi
 > end

However, this approach to debugging will likely prove futile: there are simply too many instructions executed even in most trivial programs.

A better approach might be to run the program until crash, attempt to understand what current function is doing and who calls it, and setting breakpoints appropriately.

On x86, you can often deduce function boundaries even in fully stripped executable.

Another thing you'll want to look at is strace/truss output, so you can see what system calls immediately precede the crash point.

Answer 2

1. Dissassemble the binary separately (e.g. using objdump) and consult the listing while debugging
2. Use IDA and its debugger. Much better experience IMO.

(disclaimer: I work for Hex-Rays)

Answer 3

Python scripting

This will give more flexibility than GDB scripting to achieve your crazy ideas.

The main problem here, just like with GDB scripts, is that this will likely be too slow for most applications without target hardware support, e.g.: a C hello world takes 1 minute for only 18k instructions.

gdb.py

class TraceAsm(gdb.Command):
    def __init__(self):
        super().__init__(
            'trace-asm',
            gdb.COMMAND_BREAKPOINTS,
            gdb.COMPLETE_NONE,
            False
        )
    def invoke(self, argument, from_tty):
        argv = gdb.string_to_argv(argument)
        if argv:
            gdb.write('Does not take any arguments.\n')
        else:
            done = False
            thread = gdb.inferiors()[0].threads()[0]
            last_path = None
            last_line = None
            with open('trace.tmp', 'w') as f:
                while thread.is_valid():
                    frame = gdb.selected_frame()
                    sal = frame.find_sal()
                    symtab = sal.symtab
                    if symtab:
                        path = symtab.fullname()
                        line = sal.line
                    else:
                        path = None
                        line = None
                    if path != last_path:
                        f.write("path {}{}".format(path, os.linesep))
                        last_path = path
                    if line != last_line:
                        f.write("line {}{}".format(line, os.linesep))
                        last_line = line
                    pc = frame.pc()
                    f.write("{} {} {}".format(hex(pc), frame.architecture().disassemble(pc)[0]['asm'], os.linesep))
                    gdb.execute('si', to_string=True)
TraceAsm()

GitHub upstream.

main.S

global _start
_start:
    ; Write.
    mov rax, 1
    mov rdi, 1
    mov rsi, hello_world
    mov rdx, hello_world_len
    syscall

    ; Exit.
    mov rax, 60
    mov rdi, 0
    syscall

hello_world db "hello world", 10
hello_world_len equ $ - hello_world

GitHub upstream.

Assemble and run:

as -o main.o main.S
ld -o main.out main.o
gdb -nh -batch -ex 'source ~/test/gdb.py' -ex 'starti' -ex 'trace-asm' ./main.out
cat trace.tmp

Output:

0x401000 mov    $0x1,%rax 
0x401007 mov    $0x1,%rdi 
0x40100e mov    $0x402000,%rsi 
0x401015 mov    $0xc,%rdx 
0x40101c syscall  
0x40101e mov    $0x3c,%rax 
0x401025 mov    $0x0,%rdi 
0x40102c syscall

QEMU emulation

This performed much faster than the GDB python solution, the C hello works runs instantaneously! However, the log was only 10k instructions long instead of 18k on the same executable, so it must be skipping something that normally gets run TODO understand.

E.g. in user mode simulation:

qemu-x86_64 -d in_asm ./main.out

Output:

warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
----------------
IN: 
0x0000000000401000:  mov    $0x1,%rax
0x0000000000401007:  mov    $0x1,%rdi
0x000000000040100e:  mov    $0x402000,%rsi
0x0000000000401015:  mov    $0xc,%rdx
0x000000000040101c:  syscall 

hello world
----------------
IN: 
0x000000000040101e:  mov    $0x3c,%rax
0x0000000000401025:  mov    $0x0,%rdi
0x000000000040102c:  syscall

See also.

Tested in Ubuntu 18.10, GDB 8.2, QEMU 2.12.0.