[ACCEPTED]-Python Celery versus Threading Library for running async requests-python-multithreading

Accepted answer

Score: 25

The Python interpreter is resolutely single 17 threaded due to the (in)famous global interpreter lock aka (GIL). Thus 16 threading in Python only provides parallelism 15 when computation and IO can happen simultaneously. Compute 14 bound tasks will see little benefit from 13 threading in the Python threading model, at 12 least under CPython 2 or 3.

On the other 11 hand, those limitations don't apply to multiprocessing, which 10 is what you'll be doing with a queuing system 9 like celery. You can run several worker 8 instances of Python which can execute simultaneously 7 on multicore machines, or multiple machines 6 for that matter.

If I understand your scenario 5 - an interaction on a web site kicks off 4 a long-running job - queuing is almost certainly 3 the way to go. You get true parallelism 2 and the easy option to move processing to 1 other machines.

Source: stackoverflow.com

[ACCEPTED]-Python Celery versus Threading Library for running async requests-python-multithreading

AttributeError: 'DisabledBackend' object has no attribute '_get_task_meta_for'

Scheduling thousands of one-off (non-reoccuring) tasks for near-simultaneous execution via Django-celery

Stopping celery task gracefully

How to schedule weekday-aware jobs in celery

Django celery task run at once on startup of celery server

Django: How to automatically change a field's value at the time mentioned in the same object?

Celery stop execution of a chain

Resolving circular imports in celery and django

Add n tasks to celery queue and wait for the results

Django Celery socket.error: [Errno 111] Connection refused