How can I tell where my python script is hanging?

Question

So I\'m debugging my python program and have encountered a bug that makes the program hang, as if in an infinite loop. Now, I had a problem with an infinite loop before, but

Answer 1

If your program is too big and complex to be viable for single stepping with pdb or printing every line with the trace module then you could try a trick from my days of 8-bit games programming. From Python 2.5 onwards pdb has the ability to associate code with a breakpoint by using the commands command. You can use this to print a message and continue running:

(Pdb) commands 1
(com) print "*** Breakpoint 1 ***"
(com) continue
(com) end
(Pdb)

This will print a message and carry on running when breakpoint 1 is hit. Define similar commands for a few other breakpoints.

You can use this to do a kind of binary search of your code. Attach breakpoints at key places in the code and run it until it hangs. You can tell from the last message which was the last breakpoint it hit. You can then move the other breakpoints and re-run to narrow down the place in the code where it hangs. Rinse and repeat.

Incidentally on the 8-bit micros (Commodore 64, Spectrum etc) you could poke a value into a registry location to change the colour of the border round the screen. I used to set up a few breakpoints to do this with different colours, so when the program ran it would give a psychedelic rainbow display until it hung, then the border would change to a single colour that told you what the last breakpoint was. You could also get a good feel for the relative performance of different sections of code by the amount of each colour in the rainbow. Sometimes I miss that simplicity in these new fangled "Windows" machines.

Answer 2

You could also try http://code.activestate.com/recipes/576515-debugging-a-running-python-process-by-interrupting/ . It should work as long as the Python process doesn't have signals masked, which is normally the case even if Ctrl-C doesn't work.

Answer 3

Let's assume that you are running your program as:

python YOURSCRIPT.py

Try running your program as:

python -m trace --trace YOURSCRIPT.py

And have some patience while lots of stuff is printed on the screen. If you have an infinite loop, it will go on for-ever (halting problem). If it gets stuck somewhere, then mostly you are stuck on I/O or it is a deadlock.

Answer 4

Wow ! Seems you added so much code in one go without testing it that you can't say what code was added just before program started to hang... (the most likely cause of problem).

Seriously, you should code by small steps and test each one individually (ideally doing TDD).

For your exact problem of spotting what python code is running and ctrl-c does not work, I will try a raw guess: did you used some except: catching all exceptions indistinctly. If you did so in a loop (and continue loop after managing exception), it's a very likely reason why ctrl-c does not work : it's catched by this exception. Change to except Exception: and it should not be catched any more (there is other possibilities for ctrl+c not working like thread management as another poster suggested, but I believe the above reason is more likely).

exception KeyboardInterrupt

Raised when the user hits the interrupt key (normally Control-C or Delete). 

During execution, a check for interrupts is made regularly. Interrupts typed when a built-in function input() or raw_input() is waiting for input also raise this exception. The exception inherits from BaseException so as to not be accidentally caught by code that catches Exception and thus prevent the interpreter from exiting.

Changed in version 2.5: Changed to inherit from BaseException.

Answer 5

From Python 3.3 on there is a built in faulthandler module:

import faulthandler
faulthandler.enable()

Answer 6

Nothing like the good old pdb

import pdb
pdb.run('my_method()',globals(),locals())

Then just hit (n) to go to the next command, (s) to step into. see the docs for the full reference. Follow your program step by step, and you'll probably figure it out fast enough.