[ACCEPTED]-Python Celery versus Threading Library for running async requests-python-multithreading

Accepted answer
Score: 25

The Python interpreter is resolutely single 17 threaded due to the (in)famous global interpreter lock aka (GIL). Thus 16 threading in Python only provides parallelism 15 when computation and IO can happen simultaneously. Compute 14 bound tasks will see little benefit from 13 threading in the Python threading model, at 12 least under CPython 2 or 3.

On the other 11 hand, those limitations don't apply to multiprocessing, which 10 is what you'll be doing with a queuing system 9 like celery. You can run several worker 8 instances of Python which can execute simultaneously 7 on multicore machines, or multiple machines 6 for that matter.

If I understand your scenario 5 - an interaction on a web site kicks off 4 a long-running job - queuing is almost certainly 3 the way to go. You get true parallelism 2 and the easy option to move processing to 1 other machines.

More Related questions