How do I run another script in Python without waiting for it to finish? [duplicate]

Answer 1

p = subprocess.Popen([sys.executable, '/path/to/script.py'], 
                                    stdout=subprocess.PIPE, 
                                    stderr=subprocess.STDOUT)

That will start the subprocess in background. Your script will keep running normally.

Read the documentation here.

Answer 2

subprocess.Popen is indeed what you are looking for.

Answer 3

Running this through a message queue is definitely the way to go if you're thinking about long-term scaling. Send a message to the queue who's running constantly in the background, and write job handlers to deal with the different sorts of messages.

Since you're using Django, I think Beanstalkd is a pretty good fit. Here's a pretty nice tutorial on the subject. The first comment in that article also has some good tips.

Personally I've rolled with a custom in-memory queue server written in Erlang, with Python-bindings written in C. But redis looks like it might work out as a great contender for future queuing/messaging-needs. Hope this helps!

Answer 4

Although if you find that you want to start communicating a bunch of information between the subprocess and the parent, you may want to consider a thread, or RPC framework like Twisted.

But most likely those are too heavy for your application.