“Can't start a new thread error” in Python

Question

I am running a script which extracts the information from debian packages and saves it in a database.

After extracting information from about 100 packages an error

Answer 1

Sorry to have an incomplete solution here, I don't have the rating to post in the comment section.

One thing to check is the total number of threads you have running. I have some code that checks the count of cores (with sys) and then launches threads and checks core loading to test how the OS handles thread distribution and I've learned that windows 7 (for example) seems to throw an error beyond 32 threads on an 8 (logical) core CPU. [That's on Python 2.7, 32 bit within Win 7 64-bit, etc, YMMMV]. On other machines I can get past 1,000 Threads.

So I guess the short version is: how many threads do you already have going when you get that error? You can check with

threading.active_count()

Beyond that you don't really give the threading part of the code here so I'd direct you to this excellent Python Central page.

You may also benefit from this previous Stack overflow discussion on large thread task counts and how to approach them.

Again, my apologies that this is more of a direction to look than a solution, I think that more information is likely needed to help us understand what you're running into.

Answer 2

You're using a 32bit system and running out of virtual memory. One of your libraries is likely spawning threads and not reclaiming them correctly. As a workaround, try reducing the default thread stack size with threading.stack_size.