background
As we all know, celery is a python world of distributed processing task a good helper, it appears combination gives us a powerful processing asynchronous requests, the ability to complex scenes distributed tasks, periodic tasks.
However, goose, today we want to discuss is how to better use celery, needle main point of discussion is the use of memory aspects.
django & celery & django-celery
The landlord used in the project is a combination of celery and django way versions are:
python == 2.7
celery==3.1.25
Django==1.11.7
django-celery==3.2.2
celery handle concurrent
Celery beat used to project the trigger timing task; according to business needs and, respectively, using two celery worker to handle asynchronous requests.
In the development environment, the operating system has four processors, memory to 8GB. By default, the start and two celerycelery beat worker, concurrent follows:
Can be seen that, by default, Celery will worker to initiate a corresponding number of the number of processor (4) .
celery celery allow us to control the number of concurrent worker configure 'CELERYD_CONCURRENCY'.
When the modified Celery worker to worker tasksWorker configured to: CELERYD_CONCURRENCY = 2, the number of worker follows:
We can see only two worker is activated .
celery worker on memory
There are many online posts analyzed celery worker for memory usage, found that worker does not release memory after processing tasks. However, celery provides a configuration allows the maximum number (CELERYD_MAX_TASKS_PER_CHILD) we have developed processing tasks for each worker, a worker when the number of processing tasks to develop the number of arrival, celery will destroy the worker and restart a new one.
When CELERYD_MAX_TASKS_PER_CHILD = 5, after the worker run multiple times, you can see the old worker is destroyed, the new worker is activated.
Old worker:
New worker:
In addition, when we combine the use django and celery, the need to close the Debug configuration, since open will cause a memory leak celery beat. See detailed description: https://stackoverflow.com/questions/45366680/celerybeat-process-consumes-all-os-memory