[ACCEPTED]-run subprocesses in parallel-subprocess
subprocess.call() is blocking. That means, each call must 8 wait for the child process to finish before 7 continuing.
What you want is to pass your 6 arguments to
subprocess.Popen constructor, instead. That 5 way, your child process would be started 4 without blocking.
Later on, you can join 3 these child processes together by calling 2
child_processes =  for work, filename in worklist: with io.open(filename, mode='wb') as out: p = subprocess.Popen(work, stdout=out, stderr=out) child_processes.append(p) # start this one, and immediately return to start another # now you can join them together for cp in child_processes: cp.wait() # this will block on each child process until it exits
P.S. Have you looked into Python's 1 documentation on the
I like to use
GNU Parallel (http://www.gnu.org/software/parallel/) in situations like this 22 (requires *nix), as it provides a quick 21 way to get parallelism and has many options, including 20 re-organizing the output at the end such 19 that it all flows together from each process 18 in order but not interleaved. You can also 17 specify the number you want to run at once, either 16 a specific number, or matching the number 15 of cores you have, and it will queue up 14 the rest of the commands.
subprocess.check_output with 13
shell=True to call out to
parallel using your command string. If 12 you've got a variable you want to interpolate, say 11 a list of
SQL tables you want to run your command 10 against, parallel is good at handling that 9 as well -- you can pipe in the contents 8 of a text file with the arguments.
If the 7 commands are all totally different (as opposed 6 to being variations on the same command), put 5 the complete commands in the text file that 4 you pipe into
You also don't need to do 3 anything special to wait for them to finish, as 2 the
check_output call will block until the
parallel command 1 has finished.
More Related questions