I want to know how does gdb work internally. e.g. I know a brief idea that it makes use of ptrace() system call to monitor traced program. But I want to know how it handles sign
Check out the GDB Internals Manual, which covers some of the important aspects. There's also an older PDF version of this document.
From the manual:
This document documents the internals of the GNU debugger, gdb. It includes description of gdb's key algorithms and operations, as well as the mechanisms that adapt gdb to specific hosts and targets.