I want to debug (examine DOM, use the interactive JS console, etc) part of a web application that is inside a modal dialog that was created by showModalDialog().
What I do when i'm debugging modal windows are two things.
You can include in your js code the instruction debugger; that will stop the js execution as if you had put a breakpoint.
You can also open the ie developer tools before opening the modal window and, once it's open, you can review the generated code in the script tab, adding breakpoints, pausing execution,... everything you need.