parallel processing - ipython notebook : how to parallelize external script -

parallel processing - ipython notebook : how to parallelize external script -

i'm trying utilize parallel computing ipython parallel library. have little knowledge , find doc hard read knows nil parallel computing.

funnily, tutorials found re-use illustration in doc, same explanation, point of view, useless.

basically i'd running few scripts in background executed in same time. in bash :

for my_file in $(cat list_file); python pgm.py my_file & done

but bash interpreter of ipython notebook doesn't handle background mode.

it seems solution utilize parallel library ipython.

i tried :

from ipython.parallel import client rc = client() rc.block = true dview = rc[:2] # take 2 engines

but i'm stuck. don't know how run twice (or more) same script or pgm @ same time.

thanks.

if you're trying executing external scripts in parallel, don't need utilize ipython's parallel functionality. replicating bash's parallel execution can achieved subprocess module follows:

import subprocess procs = [] in range(10): procs.append(subprocess.popen(['ls', '/users/shad/tmp/'], stdout=subprocess.pipe)) results = [] proc in procs: stdout, _ = proc.communicate() results.append(stdout)

be wary if subprocess generates lot of output, process block. if print output (results) get:

print results ['file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n']

parallel-processing ipython ipython-notebook jupyter ipython-parallel