python test development django-159.Celery asynchronous and RabbitMQ environment construction

Keywords: Python Django RabbitMQ


Celery is a Python task queue system used to handle work task assignments across threads or network nodes. It makes asynchronous task management easy.
Your application only needs to push messages to agents like RabbitMQ, and the Celery worker will pop them up and schedule tasks to execute.


Five roles of celery

  • Task s are tasks, including asynchronous tasks and celery beats
  • The Broker broker receives the message from the producer, that is, the Task, and stores the Task in the queue. The consumer of the Task is Worker. Celery itself does not provide queue services. Redis or RabbitMQ is recommended to implement queue services.
  • Worker is the unit that executes tasks. It monitors the message queue in real time. If there is a task, it obtains the task and executes it.
  • The Beat timing task scheduler sends tasks to the Broker according to the configuration timing.
  • Backend is used to store the execution results of tasks.

Environmental preparation

1.django environment v2.1.2
2. Install celery version

pip install celery==3.1.26.post2

3. Install Django celery package

pip install django-celery==3.3.1

RabbitMQ environment

Broker (rabbit MQ) is responsible for creating a task queue, assigning tasks to the task queue according to some routing rules, and then delivering tasks from the task queue to worker s
First, use docker to build rabbitMQ environment, and rabbitMQ mirrors the warehouse address Find the version with mangement, which will bring the web background management interface

Download the 3.8.0-management image

docker pull rabbitmq:3.8.0-management

Start the container and set the account admin and password 123456

docker run -d --name rabbitmq3.8 -p 5672:5672 -p 15672:15672 --hostname myRabbit -e RABBITMQ_DEFAULT_USER=admin -e RABBITMQ_DEFAULT_PASS=123456 rabbitmq:3.8.0-management

The host computer needs to open two ports: 5672 and 15672. 5672 is the port accessed by the back-end interface, and 15672 is the address of the front-end web management background page. Enter http://ip:15672 Access to web sites

Enter the previously set account admin and password 123456 to log in directly

Using Celery in Django

To use Celery in a Django project, you must first define an instance of the Celery Library (called an "application")

If you have a modern Django project layout, for example:

- proj/
  - proj/

The recommended method is to create a new proj / proj / module to define the cell instance:

import os

from celery import Celery

# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')

app = Celery('proj')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django apps.

def debug_task(self):
    print(f'Request: {self.request!r}')

Where debug_task is a test task, which can be logged out

# @app.task(bind=True)
# def debug_task(self):
#     print('Request: {0!r}'.format(self.request))

Just change this sentence in the above paragraph, 'proj' is the app name of your own django project

app = Celery('proj')

Then you need to be in your proj/proj/ module. This ensures that the application is loaded at Django startup for @ shared_ The task decorator (mentioned later) will use it:

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app

__all__ = ('celery_app',)

The above paragraph is fixed and doesn't need to be changed


Create a new under the app. If the name of the file is required, django will automatically find the file under the app

def add(x, y):
    return x + y

def mul(x, y):
    return x * y can write task functions add and mul. The most direct way to make them effective is to add app.task or shared_task this decorator

Add setting configuration add configuration

  • The broker parameter indicates the URL used to connect to the broker. Rabbitmq uses a protocol called 'amqp'. If rabbitmq runs under the default setting, celery does not need other information, as long as amqp: / /.
  • The backend parameter is optional. If you want to query task status or task execution results, it is required. The backend in the cell is used to store task results. rpc means sending the result back as an AMQP message.
#   RabbitMQ configuring BROKER_URL and backend
BROKER_URL = 'amqp://admin:123456@'

CELERY_TIMEZONE = 'Asia/Shanghai'

Create view create views

from .tasks import add, mul

def task_demo(request):
    res = add.delay(10, 20)
    print(res.task_id)  # Return task_id
    return JsonResponse({"code": 0, "res": res.task_id})

Start worker

The celery application has been installed in pip. Celery is an independent application that can start the worker

celery -A MyDjango worker -l info

Where MyDjango is your own django project name

Run log

 -------------- celery@DESKTOP-HJ487C8 v3.1.26.post2 (Cipater)
---- **** -----
--- * ***  * -- Windows-10-10.0.17134-SP0
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         yoyo:0x1ea1a96e9b0
- ** ---------- .> transport:   amqp://admin:**@
- ** ---------- .> results:     rpc://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery

  . yoyo.tasks.add
  . yoyo.tasks.mul

[2021-10-18 22:45:03,155: INFO/MainProcess] Connected to amqp://admin:**@
[2021-10-18 22:45:03,347: INFO/MainProcess] mingle: searching for neighbors
[2021-10-18 22:45:04,897: INFO/MainProcess] mingle: all alone
[2021-10-18 22:45:05,406: WARNING/MainProcess] e:\python36\lib\site-packages\celery\fixups\ 
UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2021-10-18 22:45:05,407: WARNING/MainProcess] celery@DESKTOP-HJ487C8 ready.

When running, when we see "Connected to amqp", it indicates that the connection has been successful!

shell interactive environment

Debugging and running tasks in django shell interactive environment

D:\202107django\MyDjango>python shell
Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 03:37:03) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from yoyo.tasks import add,mul
>>> from celery.result import AsyncResult
>>> res = add.delay(11, 12)
>>> res
<AsyncResult: c5ff83a4-4840-4b36-8869-5ce6081904f1>
>>> res.status
>>> res.backend
<celery.backends.redis.RedisBackend object at 0x0000015E011C3128>
>>> res.task_id
>>> get_task = AsyncResult(id=res.task_id)
>>> get_task
<AsyncResult: c5ff83a4-4840-4b36-8869-5ce6081904f1>
>>> get_task.result

res.status is to view the task status
res.task_id is the id of the acquisition task
Result.result get task results
Query the execution result asyncresult (id = res.task_id) of the task according to the ID of the task. Get the result

Posted by sargus on Fri, 22 Oct 2021 21:04:00 -0700