os.walk multiple directories at once [duplicate]

Answer 1

since nobody mentioned it, in this or the other referenced post:

http://docs.python.org/library/multiprocessing.html

>>> from multiprocessing import Pool
>>> p = Pool(5)
>>> def f(x):
...     return x*x
...
>>> p.map(f, [1,2,3])

in this case, you'd have a list of directories. the call to map would return a list of lists from each dir, you could then choose to flatten it, or keep your results clustered

def t(p):
    my_paths = []
    for path, dirs, files in os.walk(p):
        my_paths.append(dirs)


paths = ['p1','p2','etc']
p = Pool(len(paths))
dirs = p.map(t,paths)