Multiprocessing launching too many instances of Python VM

Question

I am writing some multiprocessing code (Python 2.6.4, WinXP) that spawns processes to run background tasks. In playing around with some trivial examples, I am running into

Answer 1

I don't see anything wrong with that. Works fine on Ubuntu 9.10 (Python 2.6.4).

Are you sure you don't have cron or something starting multiple copies of your script? Or that the spawned script is not calling anything that would start a new instance, for example as a side effect of import if your code runs directly on import?

Answer 2

When I run this in Linux with python2.6, I see a maximum of 4 python2.6 processes and I can't guarantee that they're all from this process. They're definitely not filling up the machine.

Need new python version? Linux/Windows difference?

Answer 3

It looks like you didn't carefully follow the guidelines in the documentation, specifically this section where it talks about "Safe importing of main module".

You need to protect your launch code with an if __name__ == '__main__': block or you'll get what you're getting, I believe.

I believe it comes down to the multiprocessing module not being able to use os.fork() as it does on Linux, where an already-running process is basically cloned in memory. On Windows (which has no such fork()) it must run a new Python interpreter and tell it to import your main module and then execute the start/run method once that's done. If you have code at "module level", unprotected by the name check, then during the import it starts the whole sequence over again, ad infinitum