Introduction

Because the project needs to include chat function, we hope to use websocket to communicate. However, dwebsocket module can not work properly under Nginx+Uwsgi deployment although it runs normally on runserver. In fact, due to the workflow of the wsgi protocol, long connections such as WebSockets are almost impossible to maintain. So we need to find a new way to realize it. Following is a description of the development environment and production environment configuration for using Redis as a django cache and websocket.

Where to Start

The original configuration is basically in accordance with Huang Chao's blog The content of this paragraph is basically consistent with this article, intercepting the content of installing virtual environment. If you have already configured Uwsgi and Nginx, just need support for websocket, you can skip this section.

mysql installation

apt install mariadb-server mariadb-client libmariadbd-dev libmysqlclient-dev
service mysql restart
service mysql status
mysql_secure_installation
Enter current password for root:[Just enter]
Set root password?[Y/n] Y

#enter your password
Remove anonymous users?[Y/n] Y
Disallow root login remotely?[Y/n] Y
Remove test database and access to it? [Y/n] Y
Reload privilege tables now?[Y/n] Y


#Thanks for using MariaDB!
service mysql restart

Install Nginx

If you run server locally, you do not need to install Nginx.

apt install nginx

Install php

apt install php php-fpm

virtual environment

apt install python3-dev
apt install python3-venv
python3 -m venv /path/to/.venv

Enter the virtual environment:

source /path/to/.venv/bin/activate

Withdrawal from the virtual environment:

deactivate

Redis Installation and Configuration

Original link

Redis installation

apt-get install redis-server

After installation, use under linux:

service redis-server start

Open redis-server.

This command can be used to detect whether redis-server is working properly:

redis-cli ping
PONG

Receive PONG instructions for normal operation.

Configure redis to use unix domain sockets

This small segment setting allows redis to use unix domain sockets instead of TCP/IP. You can choose to ignore this paragraph. This results in slightly different settings in the settings.py file of django.

vim /etc/redis/redis.conf

# Accept connections on the specified port, default is 6379.
# If port 0 is specified Redis will not listen on a TCP socket.
# port 6379

# If you want you can bind a single interface, if the bind option is not
# specified all the interfaces will listen for incoming connections.
#
# bind 127.0.0.1

# Specify the path for the unix socket that will be used to listen for
# incoming connections. There is no default, so Redis will not listen
# on a unix socket when not specified.
#
unixsocket /path/to/redis.sock
unixsocketperm 777

In fact, the above will be:

port 6379
bind 127.0.0.1

Comment it out and add:

unixsocket /path/to/redis.sock
unixsocketperm 777

Cancel the comment.
After changing conf, use:

service redis-server restart

Restart the service.

Configure Redis as the back end of django session storage and caching

In the settings.py file, set:

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "unix:///path/to/redis.sock?db=0",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
        }
    }
}

Guarantee the following in MIDDLEWARE_CLASSES:

MIDDLEWARE_CLASSES = [
    'django.middleware.cache.UpdateCacheMiddleware',    # This must be first on the list
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    ...,
    'django.middleware.cache.FetchFromCacheMiddleware', # This must be last
    ]

So far, cache can be used in code. The simplest example:

from django.core.cache import cache
example_key = "IAmKey"
example_value = cache.get(example_key)
cache.set(example_key, {"value1", 1})
cache.set("anotherKey", "hello", timeout=None)

redis is a key-value storage mechanism. The cache.get(key) function returns None when the value corresponding to the key does not exist. The cache.set(key, value) function can store data, and the optional parameter timeout determines the expiration time in seconds. Timout = 0 means immediate failure and timeout=None means never failure.

All data stored in redis can be deleted most directly:

redis-cli flushall

In addition, redis can be set to store django session data:

SESSION_ENGINE = "django.contrib.sessions.backends.cache"
SESSION_CACHE_ALIAS = "default"

django-websocket-redis

First link Official documents In which github Find a routine.
django-websocket-redis needs to configure Redis, use Redis as a message queue, and maintain connections.

install

Install using pip:

pip install django-websocket-redis

To configure

The configuration of the development environment (runserver) is different from that of the production environment (Nginx+Uwsgi). Just take this as an example:

development environment

In the development environment, redis uses the default configuration, i.e. 127.0.0.1:6379. The redis-related settings configuration is shown in the previous note.

In settings.py

INSTALLED_APPS = [
    ...
    'ws4redis',
    ...
]
TEMPLATES = [
    {
        ...,
        'OPTIONS': {
            'context_processors': [
                ...,
                'django.template.context_processors.static',
                'django.contrib.auth.context_processors.auth',
                'ws4redis.context_processors.default',
            ],
            ...,
        },
    },
]
WSGI_APPLICATION = 'ws4redis.django_runserver.application'
WS4REDIS_CONNECTION = {
    'db': 7
}
WS4REDIS_EXPIRE = 3600
WEBSOCKET_URL = '/ws/'
WS4REDIS_PREFIX = 'ws'

production environment

In production environments, redis uses unix domain socket configuration, as described earlier

In settings.py

INSTALLED_APPS = [
    ...
    'ws4redis',
    ...
]
TEMPLATES = [
    {
        ...,
        'OPTIONS': {
            'context_processors': [
                ...,
                'django.template.context_processors.static',
                'django.contrib.auth.context_processors.auth',
                'ws4redis.context_processors.default',
            ],
            ...,
        },
    },
]
WSGI_APPLICATION = 'appname.wsgi.application'
WS4REDIS_CONNECTION = {
    'unix_socket_path': '/path/to/redis.sock',
    'db': 5
}
WS4REDIS_EXPIRE = 3600
WEBSOCKET_URL = '/ws/'
WS4REDIS_PREFIX = 'ws'

Here the appname is changed to the name of the project.

Use

Official documents about the use of written more comprehensive, here only in conjunction with my project to explain some of the functions.

Part django

In my project, I only used to send messages to designated users. ws4redis can also set up broadcasting or send messages to groups.

from ws4redis.publisher import RedisPublisher
from ws4redis.redis_store import RedisMessage

redis_publisher = RedisPublisher(facility='foobar', users=['username1', 'username2'])
message = RedisMessage('Hello World')
# and somewhere else
redis_publisher.publish_message(message)

Here username 1, username 2 is actually:

from django.contrib.auth.models import User

In User.username, RedisPublisher sends messages to websocket connections logged in with these usernames.

Part js

The js section can be written in the same way as WebSocket:

var websocket = new WebSocket("ws://yourhost/ws/foobar?subscribe-user")
websocket.on.onmessage = function (data) {
    ......
}

Replace yourhost with the target address. / ws / is the WEBSOCKET_URL previously specified in settings.py, and foobar?subscribe-user is a parameter that determines which messages this connection accepts. This setting corresponds to the previous django code for receiving messages to that user. Other settings can be found in official documents.

deploy

The project deployment with django-websocket-redis configuration is different from the simple Django project. The project is deployed according to the production environment mentioned above. Installation of Nginx et al. has been mentioned before.

Uwsgi

First, make sure that openssl is installed:

apt-get install openssl

Install Uwsgi:

pip install uwsgi

If you haven't installed django in your virtual environment, you can:

pip install Django
pip install django

However, django seems to have recently updated version 2.0, which may cause your project to fail to work properly, so you can use:

pip install Django==1.11.7

Explicit version. requirements.txt for my project will be given at the end of this article.

Next, you need to set up uwsgi configuration for django and websocket respectively:

django

Create py files

vim /path/to/example/example/wsgi_django.py

import os
os.environ.update(DJANGO_SETTINGS_MODULE='example.settings')
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

Create ini file

vim /path/to/uwsgi_django.ini

[uwsgi]
socket = /path/to/django.sock
chdir = /path/to/example
wsgi-file = example/wsgi_django.py
touch-reload = /path/to/reload_django
home = /path/to/.venv/
processes = 5
chmod-socket = 777
buffer-size = 32768
master = true
daemonize = /path/to/django.log
vacuum = true 

websocket

Create py files

vim /path/to/example/example/wsgi_websocket.py

import os
import gevent.socket
import redis.connection
redis.connection.socket = gevent.socket
os.environ.update(DJANGO_SETTINGS_MODULE='example.settings')
from ws4redis.uwsgi_runserver import uWSGIWebsocketServer
application = uWSGIWebsocketServer()

Create ini file

vim /path/to/uwsgi_websocket.ini

[uwsgi]
http-socket = /path/to/websocket.sock
chdir = /path/to/example
wsgi-file = example/wsgi_websocket.py
touch-reload = /path/to/reload_websocket
home = /path/to/.venv/
processes = 2
gevent = 1000
buffer-size = 32768
chmod-socket = 777
http-websockets = true
master = true
daemonize = /path/to/websocket.log
vacuum = true

Here the touch-reload file needs to be created manually, and other files will be created automatically at run time.

Run separately:

uwsgi --ini /path/to/uwsgi_django.ini
uwsgi --ini /path/to/uwsgi_websocket.ini

When there is a file change, execute:

touch /path/to/reload_django
touch /path/to/reload_websocket

uwsgi can be restarted separately.

Nginx

Change user:

vim /etc/nginx/nginx.conf

Change the first line of www-data to your user name (mine is root).

Fill in the configuration:

sudo vim /etc/nginx/conf.d/somename.conf

server {
 listen 80;
 server_name ***.***.***.***; #Exchange it to your own IP
 charset utf-8;

client_max_body_size 75M;


location /static {
 alias /path/to/example/static;
 }


location / {
 uwsgi_pass unix:///path/to/django.sock;
 include /etc/nginx/uwsgi_params;
 }
location /ws/ {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_pass http://unix:/path/to/websocket.sock;

 }
}

Restart Nginx:

service nginx restart

So far, the deployment has been completed.

Summary of Common Commands

After the project changes, use:

touch /path/to/reload_django
touch /path/to/reload_websocket

Restart uwsgi.

By:

redis-cli command

Operate redis, such as:

redis-cli flushall

You can empty the redis database.

If necessary, adopt:

service nginx restart

Restart the nginx server.

Pass:

service redis-server restart

Restart redis server.

Pass:

ps aux|grep uwsgi
ps aux|grep redis-server
ps aux|grep nginx

You can view the service process separately and pass:

Kill-9 process number

Or:

killall -9 uwsgi
killall -9 redis-server
killall -9 nginx

Forced interruption.

After deployment according to this method, / ws / requests, common requests and related error information are recorded in / path/to/django.log and / path/to/websocket.log respectively, which can be viewed.

Common mistakes

Updates on Web Pages

After configuring redis, it may occur that there is no update information after a page is refreshed. Looking at the log, it is found that no corresponding http request has been received. This (possibly) is due to the caching mechanism of redis. By adding in settings.py:

CACHE_MIDDLEWARE_ANONYMOUS_ONLY = True
CACHE_MIDDLEWARE_SECONDS = 0

To solve this problem. The first means that only anonymous users are cached, and the second means that the cache invalidation time is set to immediately invalidate. I found that adding the first item didn't solve my problem, so I added the second one. Maybe there's a more elegant way to solve it, but I didn't find it.

Repeated display of chat messages

After configuring ws4redis in my project, when a chats with b, a goes offline, refreshes the page and logs in again, the last message from B to a will be received again. The check found that it was websocket that received the message again.

The solution is to change WS4REDIS_EXPIRE = 0 in settings.py to invalidate the message in redis immediately after it is received by the front end.

About safari

ws4redis actually works well in safari. But when using runserver, it may appear at the back end:

websocket.receive: WebSocketError Unexpected EOF while decoding header

At the front end:

WebSocket connection to 'ws://127.0.0.1:8000/ws/foobar?subscribe-user' failed: Invalid HTTP version string: HTTP/1.0

This issue was raised as an issue in github, but it has not changed. The result of the discussion is the problem of django runserver. This problem disappeared after deployment to the server.

About 502

If 502 Bad Gateway occurs when a websocket connection is established, it is likely that the Nginx configuration is incorrectly set. Check that the conf file of Nginx is correct in forwarding the request to Uwsgi.

About 500

If a websocket connection is established:

WebSocket connection to 'ws://***' failed: Error during WebSocket handshake: Unexpected response code: 500

See in the log:

you need to build uWSGI with SSL support to use the websocket handshake api function !!!

Explain that your uwsgi does not support ssl.

This is a very troublesome problem because the server was already installed with openssl when this error occurred. The simple solution is to install Uwsgi outside the virtual environment after installing openssl and execute:

killall -9 uwsgi
uwsgi --ini /path/to/uwsgi_django.ini
uwsgi --ini /path/to/uwsgi_websocket.ini

In general, this solves the problem, because the Uwsgi installed in the virtual environment may not be able to locate your openssl.

If you still can't solve this problem, A Problem on Stack Overflow More possible solutions are given.

Attachment: requirements.txt

Following is the requirements.txt of my project so far, which is not guaranteed to be all useful, but if you find any packages missing somewhere in the process of configuring according to this article, maybe you can find them here:

amqp==2.2.2
billiard==3.5.0.3
celery==4.1.0
coverage==4.4.2
Django==1.11.7
django-redis==4.8.0
django-redis-cache==1.7.1
django-redis-sessions==0.6.1
django-websocket-redis==0.5.1
flake8==3.5.0
gevent==1.2.2
greenlet==0.4.12
kombu==4.1.0
mccabe==0.6.1
mysqlclient==1.3.12
pep8-naming==0.4.1
pycodestyle==2.3.1
pycrypto==2.6.1
pyflakes==1.6.0
python-redis-lock==3.2.0
pytz==2017.3
redis==2.10.6
redis-cache==0.1.3
redis-structures==0.1.6
six==1.11.0
vine==1.1.4
vital-tools==0.1.8
wsaccel==0.6.2

Reference summary

1.Using Redis as Django's session store and cache backend

2.Websockets for Django applications using Redis as message queue

3.How to Set Up My First Server?

4.Error during WebSocket handshake: Unexpected response code: 500