A tour of Celery
What is a distributed task queue and how does Celery implement one? It turns out that distributed task queues are a type of architecture that has been around for quite some time. They are a form of master-worker architecture with a middleware layer that uses a set of queues for work requests (that is, the task queues) and a queue, or a storage area, to hold the results (that is, the result backend).
The master process (also called a client or producer) puts work requests (that is, tasks) into one of the task queues and fetches results from the result backend. Worker processes, on the other hand, subscribe to some or all of the task queues to know what work to perform and put their results (if any) into the result backend.
This is ...
Get Distributed Computing with Python now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
