Comments on: Python – parallelizing CPU-bound tasks with multiprocessing

By: jc RAGRIS

jc RAGRIS — Thu, 09 May 2013 08:21:55 +0000

For a year I follow with interest the project ” Supercomputing for Everyone “:
http://www.parallella.org/
The member of the community are either Developing Support for Python and others language!
I googled the words ” Python and parallel processing ” and I stumbled upon your site.
I just tested your algorithm on my PC (windows 7 with I7 processor and python 2.7)
With the proposed changes from E.A.Neonakis, the program works perfectly.
I verified the proper operation of the parallelization with the task manager in Windows, and miracle, the work is perfectly balanced between the 8 units of the processor.
With the processing algorithm in series only one unit works!
For 2 million numbers:
Algorithm series: 22 seconds
Parallel algorithm: 9.2 seconds
Conclusion: your algorithm works perfectly.
Congratulations for your article and your work.
And urges the arrival of the electronic board of the firm Parrallella (only $ 100)!

By: Joseph

Joseph — Fri, 08 Feb 2013 01:07:42 +0000

E.A. Neonakis -> Unfortunately, I beat my head against this last snippet of code for about 2 hours before I found your comment. without multiprocessing. pre-pending the Queue() creation, the Queue was not being shared between processes. I could see the values being put in the queue in the worker() threads but the (what turned out to be separate) queue would remain empty from the mp_factorizer() context.

By: neo

neo — Mon, 14 Jan 2013 15:10:18 +0000

Thanks for the guide. Thanks also to Neonakis, couldn’t have gotten it to work quickly on windows without his additions.

By: E.A.Neonakis

E.A.Neonakis — Fri, 12 Oct 2012 07:56:56 +0000

Thank you for this. Please allow me to comment on the modifications necessary to run under WIndows XP:
1. The worker function must be picklable, so move it outside the mp_factorizer to the top level of the module
2. The module will be imported by the parent proceses, so the code calling mp_factorizer must be guarded inside if __name == ‘__main__’:
3. The multiprocessing Queue implementation should be used, so out_q = Queue() should be replaced by out_q = multiprocessing.Queue()

By: slater

slater — Fri, 21 Sep 2012 06:28:04 +0000

Nice guide to multiprocessing and comparison with threading. Thank you.

By: eliben

eliben — Tue, 08 May 2012 05:39:23 +0000

prateek,

Not sure what you want me to fix. Anyhow, I’m not responsible to keep the DZone mirror updated.

I think I tried having a separate Q for each sub-process, but it didn’t matter in terms of performance so I just went with a single Q for simplicity. You can run your own benchmark to compare the two approaches.

By: prateek

prateek — Tue, 08 May 2012 03:28:22 +0000

http://python.dzone.com/articles/parallelizing-cpu-bound-tasks

Could you correct this information on this website. I was confused because i read your article here and assumed you created one queue per process.

Also does this implies that if I create a shared queue per process (which is shared between the process itself and the producer/feeder thread) and put the task on these per process shared queues and finally collect results from them , i can get even better speedup due to lesser contention as compared to one shared queue

Thanks for your prompt reply

By: eliben

eliben — Tue, 08 May 2012 03:01:58 +0000

prateek,

No, I just create a single Queue and pass it around to all sub-processes. See this quote from the article:

The one I’ve chosen is to simply create a Queue, and let each worker process put a result dictionary into it.

By: prateek

prateek — Tue, 08 May 2012 02:42:15 +0000

I am new to python , and am slightly confused by the code for multiprocessing. You said that you create an output queue for each of your process but to me it seems like you just create a shared out_q , that you pass as argument to all the processes and then collect the result from the same shared queue .
Could you please give more details on what is shared output queue for all processes and a queue for each processes.
I am confused about :
for i in range(nprocs):
resultdict.update(out_q.get())

how does this piece of code can collect results from queue from different processes unless it is a shared queue between every process.

By: eliben

eliben — Sun, 29 Jan 2012 18:58:23 +0000

Antoine,

Yep, thanks to your new GIL implementation! Well done there Dave Beazley covers it in one of the presentations on the page I linked to.