Wait for child using os.system

Question

I use a lot of os.system calls to create background processes inside a for loop. How can I wait for all the background processes to end ?

os.wait tells me there are no child process.

ps: I am using Solaris

here is my code :

#!/usr/bin/python
import subprocess
import os

pids = []
NB_PROC=30

for i in xrange(NB_PROC):
        p = subprocess.Popen("(time wget http://site.com/test.php 2>&1 | grep real )&", shell=True)
        pids.insert(0,p)
        p = subprocess.Popen("(time wget http://site.com/test.php 2>&1 | grep real )&", shell=True)
        pids.insert(0,p)

for i in xrange(NB_PROC*2):
        pids[i].wait()
os.system("rm test.php*")

Answer 1

Normally, os.system() returns when the child process is finished. So there is indeed nothing for os.wait() to do. It is equivalent to subprocess.call().

Use subprocess.Popen() to create background processes, and then the wait() or poll() methods of the Popen objects to wait for them to quit.

By default, Popen does not spawn a shell but executes the program directly. This saves resources and prevents possible shell injection attacks.

According to the documentation for os.system():

The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function

If you want to do multiple jobs in parallel, consider using multiprocessing, especially the Pool object. It takes care of a lot of the details of farming work out over several processes.

Edit: Timing the execution of a program;

import time
import subprocess

t1 = time.clock()
t2 = time.clock()
overhead = t2-t1

t1 = time.clock()
subprocess.call(['wget', 'http://site.com/test.php'])
t2 = time.clock()
print 'elapsed time: {:.3f} seconds.'.format(t2-t1-overhead)

Answer 2

the solution was indeed in the subprocess module

#!/usr/bin/python
import subprocess
import os

pids = []
NB_PROC=4
cmd="(time wget http://site.com/test.php 2>&1 | grep elapsed | cut -d ' ' -f 3)"

for i in xrange(NB_PROC):
    p = subprocess.Popen(cmd,stdin=None,stdout=None, shell=True)
    pids.insert(0,p)
    print "request %d processed" % (i+1)


for i in xrange(NB_PROC):
    pids[i].wait()
os.system("rm test.php*")

switched to debian in the process but for some reason sometimes the scripts hangs while sometimes it just runs fine