[ACCEPTED]-run subprocesses in parallel-subprocess

Accepted answer

Score: 12

subprocess.call() is blocking. That means, each call must 8 wait for the child process to finish before 7 continuing.

What you want is to pass your 6 arguments to subprocess.Popen constructor, instead. That 5 way, your child process would be started 4 without blocking.

Later on, you can join 3 these child processes together by calling 2 Popen.communicate() or Popen.wait().

child_processes = []
for work, filename in worklist:
    with io.open(filename, mode='wb') as out:
        p = subprocess.Popen(work, stdout=out, stderr=out)
        child_processes.append(p)    # start this one, and immediately return to start another

# now you can join them together
for cp in child_processes:
    cp.wait()                         # this will block on each child process until it exits

P.S. Have you looked into Python's 1 documentation on the subprocess module?

Score: 2

I like to use GNU Parallel (http://www.gnu.org/software/parallel/) in situations like this 22 (requires *nix), as it provides a quick 21 way to get parallelism and has many options, including 20 re-organizing the output at the end such 19 that it all flows together from each process 18 in order but not interleaved. You can also 17 specify the number you want to run at once, either 16 a specific number, or matching the number 15 of cores you have, and it will queue up 14 the rest of the commands.

Just use subprocess.check_output with 13 shell=True to call out to parallel using your command string. If 12 you've got a variable you want to interpolate, say 11 a list of SQL tables you want to run your command 10 against, parallel is good at handling that 9 as well -- you can pipe in the contents 8 of a text file with the arguments.

If the 7 commands are all totally different (as opposed 6 to being variations on the same command), put 5 the complete commands in the text file that 4 you pipe into parallel.

You also don't need to do 3 anything special to wait for them to finish, as 2 the check_output call will block until the parallel command 1 has finished.

Source: stackoverflow.com

[ACCEPTED]-run subprocesses in parallel-subprocess

How can I find out why subprocess.Popen wait() waits forever if stdout=PIPE?

Python memory allocation error using subprocess.Popen

Why does shell=True eat my subprocess.Popen stdout?

Subprocess.Popen: cloning stdout and stderr both to terminal and variables

Fire and forget" a process from a Python script

Cannot find the file specified when using subprocess.call('dir', shell=True) in Python

How to get environment from a subprocess?

pipe large amount of data to stdin while using subprocess.Popen

Wait the end of subprocesses with multiple parallel jobs

Module subprocess has no attribute 'STARTF_USESHOWWINDOW'