Debugger in Python freezes over own built modules

问题

I think this question is addressed here: Python spyder debug freezes with circular importing

But I didn't quite get a solution.

First time using the Debugger in Python. I am using IDLE. The program runs fine otherwise, but when run through the Debugger it freezes and I have to kill the program. The problem, I have figured out, is that I import scripts that I have created, that I use mostly as databases. So it's something like this.

from ownScript import array

where array is just a list of lists, like a CSV file. I find this to be a quite straightforward way to use data locally, even if I will surely implement some other way in the future.

If I use "over" it just freezes, but if I "step" my way to the error I get this:

Traceback (most recent call last):
  File "/Users/Adrian/Documents/testModuleCaller.py", line 5, in <module>
    import testModule
  File "<frozen importlib._bootstrap>", line 980, in _find_and_load
  File "<frozen importlib._bootstrap>", line 148, in __enter__
  File "<frozen importlib._bootstrap>", line 174, in _get_module_lock
  File "<frozen importlib._bootstrap>", line 59, in __init__
  File "<frozen importlib._bootstrap>", line 59, in __init__
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/bdb.py", line 88, in trace_dispatch
    return self.dispatch_line(frame)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/bdb.py", line 112, in dispatch_line
    self.user_line(frame)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/idlelib/debugger.py", line 24, in user_line
    self.gui.interaction(message, frame)
AttributeError: '_ModuleLock' object has no attribute 'name'

I understand that calling a script and importing a module are different things. I checked other modules, such as re and sys, and tried to copy some code to avoid the error, but without success. I also checked importlib/_bootstrap.py but didn't understand what I can do about _ModuleLock.

As I see it, I either modify my "module" to do the same job as other modules, or I find another way to work with data in my programs, or I use another interpreter.

EDIT: So I tested a little: by shortening the database (the list that I am importing) everything worked fine. So I tested with the original size, and it doesn't freeze, but it takes a lot of time, so that it becomes impractical. Like when you try to print very long lists and IDLE slows down a lot.

In summary:

the debugger doesn't freeze, but it's very slow when importing long datasets
the error that I posted I actually get from any other module too, such as "datetime"

Any solution to increase the speed?

回答1:

I figured out the problem is the size of the modules I import

The solution is, quite simply: don't use python modules as databases - at least not on huge datasets.

Huge datasets will always be a PITA to deal with (yes, they take long to load / transfert, and yes they easily eat up all your ram), but there's a reason we have proper databases systems (relational or not) - and not only for storage, relational databases are designed for data processing too and are usually quite optimized.

Also if you have to work on huge datasets, be ready to rethink some of your code so it can use lazy evalutions (generators / lazy iterators), parallelization (multiprocessing, map/reduce, ...), memoization where it makes sense, etc.

回答2:

The traceback is similar to the one in https://bugs.python.org/issue33065. I believe both are due to a bug in a line of code in idlelib that does not appear in the tracebacks. I just applied a fix that will be in the next releases of 3.8, 3.9, and 3.10. The way to fix current releases in in https://stackoverflow.com/a/61891185/722804. If the problem persists, I would like more details.

来源：https://stackoverflow.com/questions/54785596/debugger-in-python-freezes-over-own-built-modules

标签

python

debugging

python-module

python-idle