Django uses celery and NGINX to generate static pages for performance optimization

Keywords: Python Nginx Celery Redis Django

Performance optimization principle:

When we want to return a page to client browser, we need to query data in database and render the data and basic page template to form a page to return to the client. But if every user visits the home page once, it will undoubtedly bring great performance problems to database query when the amount of visits is large. To solve this problem, we can return an early rendered static home page to the unregistered user (a page that calls cached data and personal data rendering to the logged user), so as to improve the performance of the website.


Using celery to generate static home pages

Generate static page principle:

On the basis of a basic template for static homepage, get data, load the basic template with django loader, and render the page to generate several pages.

Install celery

pip install celery

Configure settings files for redis

# diango Cache configuration of
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",

Prepare a homepage static template file static_base.html

{# Home page registration login #}
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "">
<html xmlns="" xml:lang="en">
{% load staticfiles %}
    <meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
    {# Page Title Content Block #}
    <title>{% block title %}{% endblock title %}</title>
    <link rel="stylesheet" type="text/css" href="{% static 'css/reset.css' %}">
    <link rel="stylesheet" type="text/css" href="{% static 'css/main.css' %}">
    {# Introducing File Blocks at the Top of Web Pages #}
    {% block topfiles %}{% endblock topfiles %}
{# Welcome Block at the Top of Web Page #}
{% block header_con %}
    <div class="header_con">
        <div class="header">
            <div class="welcome fl">Welcome to Mall!</div>
            <div class="fr">
                <div class="login_btn fl">
                    <a href="{% url 'user:login' %}">Sign in</a>
                    <a href="{% url 'user:register' %}">register</a>
                <div class="user_link fl">
                    <a href="{% url 'user:user' %}">User center</a>
                    <a href="cart.html">My shopping cart</a>
                    <a href="{% url 'user:order' %}">My order</a>
{% endblock header_con %}

{# Search Block at the Top of Web Page #}
{% block search_bar %}
    <div class="search_bar clearfix">
        <a href="index.html" class="logo fl"><img src="{% static 'images/logo.png' %}"></a>
        <div class="search_con fl">
            <input type="text" class="input_text fl" name="" placeholder="Search merchandise">
            <input type="button" class="input_btn fr" name="" value="search">
        <div class="guest_cart fr">
            <a href="#" class="cart_name fl">My shopping cart</a>
            <div class="goods_count fl" id="show_count">{{ cart_count }}</div>
{% endblock search_bar %}

{# Main Content Block of Website #}
{% block body %}{% endblock body %}

    <div class="footer">
        <div class="foot_link">
            <a href="#">About us</a>
            <a href="#">Contact us</a>
            <a href="#">Recruitment of talents</a>
            <a href="#">Friendship link</a>        
        <p>CopyRight © 2016 Shanghai Shangcheng Information Technology Co., Ltd. All Rights Reserved</p>
        <p>Tel: 010-****888    Beijing ICP prepare*******8 Number</p>
    {# Bottom of webpage html Element block #}
    {% block bottom %}{% endblock bottom %}
    {# Introducing File Blocks at the Bottom of Web Pages #}
    {% block bottomfiles %}{% endblock bottomfiles %}

On the basis of the static template file of the home page, a static_index.html file of the home page is inherited to facilitate celery to get the database file and render it.

{% extends 'static_base.html' %}
{% load staticfiles %}
{% block title %}home page{% endblock title %}
{% block topfiles %}
    <script type="text/javascript" src="{% static 'js/jquery-1.12.4.min.js' %}"></script>
    <script type="text/javascript" src="{% static 'js/jquery-ui.min.js' %}"></script>
    <script type="text/javascript" src="{% static 'js/slide.js' %}"></script>
{% endblock topfiles %}
{% block body %}
    <div class="navbar_con">
        <div class="navbar">
            <h1 class="fl">Classification of all commodities</h1>
            <ul class="navlist fl">
                <li><a href="">home page</a></li>
                <li class="interval">|</li>
                <li><a href="">Mobile phone fresh</a></li>
                <li class="interval">|</li>
                <li><a href="">Luck draw</a></li>

    <div class="center_con clearfix">
        <ul class="subnav fl">
            {% for type in types %}
                <li><a href="#model0{{ forloop.counter }}" class="{{ type.logo }}">{{ }}</a></li>
            {% endfor %}
        <div class="slide fl">
            <ul class="slide_pics">
                {% for banner in goods_banners  %}
                    <li><a href="#"><img src="{{ banner.image.url }}" alt="Slide"></a></li>
                {% endfor %}
            <div class="prev"></div>
            <div class="next"></div>
            <ul class="points"></ul>
        <div class="adv fl">
            {% for banner in promotion_banners %}
                <a href="{{ banner.url }}"><img src="{{ banner.image.url }}"></a>
            {% endfor %}

    {% for type in types %}
    <div class="list_model">
        <div class="list_title clearfix">
            <h3 class="fl" id="model0{{ forloop.counter }}">{{ }}</h3>
            <div class="subtitle fl">
                {% for banner in type.title_banners %}
                    <a href="#">{{ }}</a>
                {% endfor %}
            <a href="#" class="goods_more fr" id="fruit_more">View more ></a>

        <div class="goods_con clearfix">
            <div class="goods_banner fl"><img src="{{ type.image.url }}"></div>
            <ul class="goods_list fl">
                {% for banner in type.image_banners %}
                    <h4><a href="#">{{ }}</a></h4>
                    <a href="#"><img src="{{ banner.sku.image.url }}"></a>
                    <div class="prize">¥ {{ banner.sku.price }}</div>
                {% endfor %}
    {% endfor %}
{% endblock body %}

Create a new celery_tasks folder under the project, create file in the folder, and write tasks file.

from django.conf import settings
from celery import Celery
from django.template import loader

# Add these sentences at the end of the task handler
import os
# import django
# os.environ.setdefault("DJANGO_SETTINGS_MODULE", "shoppingmall.settings")
# django.setup()

# These classes should be placed in django The environment initializes the following four sentences
from goods.models import GoodsType, IndexGoodsBanner, IndexPromotionBanner, IndexTypeGoodsBanner

# Create a Celery Instance object of class
app = Celery('celery_tasks.tasks', broker='redis://')

def generate_static_index_html():
    '''Generate homepage static pages'''
    # Get information about the types of goods
    types = GoodsType.objects.all()
    # Get the Home Rotary Commodity Information
    goods_banners = IndexGoodsBanner.objects.all().order_by('index')
    # Get Home Page Promotion Information
    promotion_banners = IndexPromotionBanner.objects.all().order_by('index')
    # Get information about classified merchandise display on home page
    for type in types:  # GoodsType
        # Obtain type Picture Display Information of Categorized Goods on Category Home Page
        image_banners = IndexTypeGoodsBanner.objects.filter(type=type, display_type=1).order_by('index')
        # Obtain type Text Display Information of Categorized Goods on Category Home Page
        title_banners = IndexTypeGoodsBanner.objects.filter(type=type, display_type=0).order_by('index')
        # Dynamically give type Adding attributes to save image display information and text display information of classified goods on the home page
        type.image_banners = image_banners
        type.title_banners = title_banners

    # Organization Template Context
    context = {
                'types': types,
                'goods_banners': goods_banners,
                'promotion_banners': promotion_banners

    # Using templates
    # 1.Load template files,Return template object
    temp = loader.get_template('static_index.html')
    # 2.Template rendering
    static_index_html = temp.render(context)

    # Generate Home Page Corresponding Static Files
    save_path = os.path.join(settings.BASE_DIR, 'static/index.html')
    with open(save_path, 'w', encoding='utf-8') as f:

Open redis service

E:\>cd E:\YifChanSoft\Database\Redis\RedisSoft\Redis-x64-3.2.100

E:\YifChanSoft\Database\Redis\RedisSoft\Redis-x64-3.2.100>redis-server --service-install --loglevel verbose

E:\YifChanSoft\Database\Redis\RedisSoft\Redis-x64-3.2.100>redis-cli> select 8
OK[8]> keys *
1) "_kombu.binding.celery"
2) "_kombu.binding.celery.pidbox"[8]>

Open redis service screenshot

Put a copy of the project code somewhere, enter it, and start tasks'worker mode.
Note that the tasks file used as the worker's code should have the initialized code to start django in advance, otherwise the worker can't call conf information;

That is to say, there should be the following contents

# Add these sentences at the end of the task handler
import os
import django
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "shoppingmall.settings")

To solve the problem of celery 4.x running on win10, install eventlet

pip install eventlet

Enter the project code where the replication is used as a celery worker

Open worker mode

celery -A celery_tasks.tasks worker -l info -P eventlet

Open screenshot of worker mode

If so, delete the index.html file in the celery code file static;
Initially call generate_static_index_html.delay() to verify the generation of index.html;

from celery_tasks.tasks import generate_static_index_html

Verify screenshots


You can see that index.html is generated under the static folder under the project.

Open the project and enter the browser to see the generated static home page; because there is no data in the database, the page is relatively empty.


Installation of NGINX

Reference course: 

1. Download nginx: 

2. Decompress nginx package

After downloading, put it in the appropriate directory. After decompression, it is as follows

3. Use the cmd command, enter the decompression directory where nginx is located, and install nginx with the following commands;

start nginx.exe

Installation screenshots

When the installation is complete, we can see the nginx task in the task manager, as shown in the figure

So far, nginx is installed.


nginx command

start nginx.exe  # open nginx
nginx -s reload  # Restart
nginx -s stop  # Stop it nginx
nginx -s quit  # Sign out nginx


Providing static home pages using NGINX

Modify nginx configuration

Find the configuration file for nginx. As shown in the figure below, we copied a source file and renamed it nginx_origin.conf for the convenience of other projects in the future.

Open the nginx.conf file with the editor and modify the configuration file as follows:

location /static {
    alias E:/Pycharm/Pycharm_save/cp15/18Django_fresh2/step206/shoppingmall206/static/;

location / {
    # root   html;
    root   E:/Pycharm/Pycharm_save/cp15/18Django_fresh2/step206/shoppingmall206/static/;
    index  index.html index.htm;

Configuration screenshots

Note that the address should be the absolute path address where you use celery's project, and that slashes / not backslashes should be used between addresses, otherwise errors will be reported.


After the configuration has been modified and saved, we use a command to restart nginx

nginx -s reload

Then, we open the browser and enter one of the two links to see the main page of the project.  # Note that there must be one after that./,Otherwise it will enter nginx Default interface

Screenshots of the main page of the project


Screenshots of the cmd command of nginx, where errors are due to the use of backslashes from the win10 directory

The relationship between Django and celery can be understood as juxtaposition. Before them, there was actually an nginx server responsible for scheduling.
Generally, when users access directly, we use nginx scheduling to return static pages in celery's nginx.
When the user visits, we return to IndexView which calls Django website.

We use nginx to configure websites when they are online.


Restructuring static pages when background data is modified


sava_model and delete_model methods under admin.ModelAdmin are called to update data when the database data changes, and we need to regenerate static pages when the data changes.

Therefore, we can customize a class to inherit admin.ModelAdmin, rewrite the method of updating and deleting data, call the method of updating and deleting the parent class, and then call the method in celery to regenerate the static homepage.



To configure the static page to be regenerated when the data of a table changes, we need to define an xxxModelAdmin class for the table, inherit from admin.ModelAdmin and rewrite the methods therein, and the table should also inherit the xxModelAdmin class when registered in admin.

Because many tables need to be configured in this way, and the code in the class is the same, we can extract a BaseModelAdmin class, write the updated code to call back the static page, and then let the tables that need to be modified inherit the class.

Write the following code in the file of the corresponding application on the home page

from django.contrib import admin
# from django.core.cache import cache
from goods.models import GoodsType, GoodsSKU, Goods, GoodsImage, IndexGoodsBanner, IndexTypeGoodsBanner, IndexPromotionBanner
from celery_tasks.tasks import generate_static_index_html

class BaseModelAdmin(admin.ModelAdmin):
    """When the background database data changes celery Restructuring static home page"""
    def save_model(self, request, obj, form, change):
        """Called when updating or adding data"""
        super().save_model(request, obj, form, change)
        # Send out the task, let's celery worker Restructuring static home page

        # Clear Cached Data on Home Page
        # cache.delete("index_page_data")

    def delete_model(self, request, obj):
        """Called when data is deleted"""
        super().delete_model(request, obj)

        # Clear Cached Data on Home Page
        # cache.delete("index_page_data")

class GoodsTypeAdmin(BaseModelAdmin):

class IndexGoodsBannerAdmin(BaseModelAdmin):

class IndexTypeGoodsBannerAdmin(BaseModelAdmin):

class IndexPromotionBannerAdmin(BaseModelAdmin):
    pass, GoodsTypeAdmin), IndexGoodsBannerAdmin), IndexTypeGoodsBannerAdmin), IndexPromotionBannerAdmin)

So far, when we update the data in admin background, we will regenerate the static homepage. You can try it on your own.~

Posted by dshevnock on Mon, 07 Oct 2019 13:28:29 -0700