Golang practice record: using gin framework to realize forwarding function: forwarding using nginx

Keywords: Go Docker Nginx

Recently, we need to implement a gadget to forward post requests to the specified back-end service. Because we always want to learn the gin framework, we use this framework to try. It is expected to produce several articles. This paper studies how to test the forwarding tool by using nginx container and back-end service.

summary

Back end service

The previous article has implemented a simple back-end service response function, which is not listed here. It should be noted that since the forwarding and back-end of this series use the same set of codes, if you want to start the back-end service in the program, you must modify the executable name of the back-end service. Refer to the previous code.

nginx container startup and restart

This article uses the image centos/nginx-116-centos7 for testing. Simple startup is as follows:

docker run -itd --name nginx -p 84:8080 -v $PWD:/opt/bin centos/nginx-116-centos7 bash

In practice, docker compose is used for startup. The docker-compose.yml file is as follows:

version: '2'

services:
  nginx:
    image: centos/nginx-116-centos7
    container_name: nginx
    command: /bin/bash -c "/home/latelee/bin/run.sh"
    volumes:
      - $PWD/bin:/home/latelee/bin
      - $PWD/nginx/log:/var/opt/rh/rh-nginx116/log/nginx
      - $PWD/nginx/etc:/etc/nginx
      - /etc/localtime:/etc/localtime
    #environment:
    #  - ORACLE_HOME=/work/instantclient_12_1
    #  - TNS_ADMIN=$ORACLE_HOME
    ports:
      - 8080:8080
      - 8090:8090

Note: the back-end service programs (and startup scripts) and nginx configuration files are placed in the host directory because it is easier to update.

In order to run customized commands when the container is started, you need to close the background execution of nginx service. The start script run.sh is as follows:

#!/bin/bash

echo "run..."
cd /home/latelee/bin

nginx -g "daemon off;"&

sleep 1
./httpforward.exe -p 8090

The official docker describes how to start nginx in the container:

If you add a custom CMD in the Dockerfile, be sure to include -g daemon off; in the CMD in order for nginx to stay in the foreground, so that Docker can track the process properly (otherwise your container will stop immediately after starting)!

The main idea is that when we customize the startup command, - g daemon off must be added, otherwise the container will exit automatically after the custom command is started.

Start container:

docker-compose up -d

View log:

docker-compose logs -f

By default, nginx is already started. If the configuration is modified. You can restart the container or within the container. The restart command is nginx -s reload. nginx -s only supports four parameters: stop, quit, reopen and reload, which are called signal s in the official documents. If you execute nginx -s stop to stop nginx and restart again, you will be prompted with nginx: [error] open() "/var/opt/rh/rh-nginx116/run/nginx/nginx.pid" failed (2: No such file or directory). Therefore, after modifying the nginx configuration, you can use reload.

Some tests of nginx forwarding

In the author's environment, whether in the container or on the physical machine, the nginx configuration file is / etc/nginx/nginx.conf (Note: in fact, nginx will also read the / etc/nginx/conf.d directory, but it is not within the scope of this article). After modifying this file, you must execute nginx -s reload to restart nginx.
The configuration of location is generally as follows:

location /fee/9000 {
        proxy_pass http://127.0.0.1:9000;
        proxy_set_header Host $proxy_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

If the URL of the request sent to the nginx server is / fee/9000, it will be forwarded to the local 9000 port service. (Note: / fee/9000 can continue to add suffix URL, but it is not the scope of this discussion). Other forwards are similar.
After testing, proxy is used directly_ The pass field specifies the back-end service address. As follows:

location /fee/9000 {
        proxy_pass http://127.0.0.1:9000;
}

It is found in the test that when the above forwarding rules are used, the back-end program must implement the response of / fee / 9000, that is, it can request normally in the container http://127.0.0.1:9000/fee/9000 . If not, 404 will be prompted. The back end of this article cannot respond to URL s with suffixes.

After investigation, adding a slash / after the back-end program URL can solve the problem, as follows:

location /fee/9000 {
        proxy_pass http://127.0.0.1:9000/;
}

The comparison is as follows:

Without slash: http://127.0.0.1:8080/fee/9000  --> http://127.0.0.1:9000/fee/9000
 Add slash: http://127.0.0.1:8080/fee/9000  --> http://127.0.0.1:9000/

Add slash extension: http://127.0.0.1:8080/fee/9000/foo  --> http://127.0.0.1:9000/foo

For the author's application, the URL of the request back-end program must be http://127.0.0.1:9000/ No additional suffixes can be added. Because the research on nginx is not deep, it will not be discussed for the time being.

experimental result

Request command:

$ curl http://192.168.28.11:8090/fee/test -X POST -F  "file=@sample.json"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   434  100    98  100   336   3920  13440 --:--:-- --:--:-- --:--:-- 18083{"code":0,"data":{"result":"ok in back end server, port: 9001 info: run in port 9001"},"msg":"ok"}

Forwarding tool:

nginx    | got url:  http://127.0.0.1:8080/fee/9001
nginx    | [GIN] 2021/09/14 - 01:32:17 | 200 |   10.002826ms |    192.168.28.5 | POST     "/fee/test"

nginx log:

==> nginx/log/access.log <==
127.0.0.1 - - [14/Sep/2021:01:32:17 +0800] "POST /fee/9001 HTTP/1.1" 200 98 "-" "Go-http-client/1.1" "-"

Summary

In this paper, the self-made forwarding tool needs to manage and configure the back-end programs and nginx, that is, start several back-end programs, allocate the port number (starting from 9000) and URL according to the rules, write them to the nginx configuration file, and finally restart the nginx service—— Of course, these are simpler methods.
In starting the back-end program, because the author uses the same code, it takes some time to troubleshoot the problem.
When testing nginx forwarding URL, it also took a lot of time (about two late nights).

In the follow-up, the author will study how to self implement the load balancing algorithm—— It has been planned to do this for more than half a year. Take this opportunity to take some time to do it.

reference resources

Discussion on Forwarding URL https://stackoverflow.com/questions/42997684/nginx-on-docker-doesnt-work-with-location-url
Nginx image: https://hub.docker.com/_/nginx

Li Chi -
layout: post
title: Golang practice record: using gin framework to realize forwarding function: forwarding using nginx
urlname: golang-notes-print-structure
tags:

golang
categories:
golang
date: 2021-09-20 02:01:00
photos:
gallery/tech/go1.jpg

summary

The forwarding tool is relatively simple to implement, but in order to verify, other tools need to be used. This paper compiles the back-end program and uses nginx to forward multiple back-end programs.
For ease of understanding, an example diagram is given below.
Forwarding without nginx is shown in the following figure:
[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-p3qhyKp3-1632075139423)(2021-09-20-Golang practice record: using gin framework to realize forwarding function: using nginx forwarding / 1.png)]

Use nginx to realize forwarding, as shown in the following figure:
[the external chain picture transfer fails, and the source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-z0edIDjW-1632075139424)(2021-09-20-Golang practice record: using gin framework to realize forwarding function: using nginx forwarding / 2.png)]

Notice the URL changes in the figure.

Back end service

nginx container startup and restart

This article uses the image centos/nginx-116-centos7 for testing. Simple startup is as follows:

docker run -itd --name nginx -p 84:8080 -v $PWD:/opt/bin centos/nginx-116-centos7 bash

In practice, docker compose is used for startup. The docker-compose.yml file is as follows:

version: '2'

services:
  nginx:
    image: centos/nginx-116-centos7
    container_name: nginx
    command: /bin/bash -c "/home/latelee/bin/run.sh"
    volumes:
      - $PWD/bin:/home/latelee/bin
      - $PWD/nginx/log:/var/opt/rh/rh-nginx116/log/nginx
      - $PWD/nginx/etc:/etc/nginx
      - /etc/localtime:/etc/localtime
    #environment:
    #  - ORACLE_HOME=/work/instantclient_12_1
    #  - TNS_ADMIN=$ORACLE_HOME
    ports:
      - 8080:8080
      - 8090:8090

Note: the back-end service programs (and startup scripts) and nginx configuration files are placed in the host directory because it is easier to update.

In order to run customized commands when the container is started, you need to close the background execution of nginx service. The start script run.sh is as follows:

#!/bin/bash

echo "run..."
cd /home/latelee/bin

nginx -g "daemon off;"&

sleep 1
./httpforward.exe -p 8090

The official docker describes how to start nginx in the container:

The main idea is that when we customize the startup command, - g daemon off must be added, otherwise the container will exit automatically after the custom command is started.

Start container:

docker-compose up -d

View log:

docker-compose logs -f

Some tests of nginx forwarding

location /fee/9000 {
        proxy_pass http://127.0.0.1:9000;
        proxy_set_header Host $proxy_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

location /fee/9000 {
        proxy_pass http://127.0.0.1:9000;
}

After investigation, adding a slash / after the back-end program URL can solve the problem, as follows:

location /fee/9000 {
        proxy_pass http://127.0.0.1:9000/;
}

The comparison is as follows:

Without slash: http://127.0.0.1:8080/fee/9000  --> http://127.0.0.1:9000/fee/9000
 Add slash: http://127.0.0.1:8080/fee/9000  --> http://127.0.0.1:9000/

Add slash extension: http://127.0.0.1:8080/fee/9000/foo  --> http://127.0.0.1:9000/foo

experimental result

Request command:

$ curl http://192.168.28.11:8090/fee/test -X POST -F  "file=@sample.json"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   434  100    98  100   336   3920  13440 --:--:-- --:--:-- --:--:-- 18083{"code":0,"data":{"result":"ok in back end server, port: 9001 info: run in port 9001"},"msg":"ok"}

Forwarding tool:

nginx    | got url:  http://127.0.0.1:8080/fee/9001
nginx    | [GIN] 2021/09/14 - 01:32:17 | 200 |   10.002826ms |    192.168.28.5 | POST     "/fee/test"

nginx log:

==> nginx/log/access.log <==
127.0.0.1 - - [14/Sep/2021:01:32:17 +0800] "POST /fee/9001 HTTP/1.1" 200 98 "-" "Go-http-client/1.1" "-"

Summary

reference resources

Discussion on Forwarding URL https://stackoverflow.com/questions/42997684/nginx-on-docker-doesnt-work-with-location-url
Nginx image: https://hub.docker.com/_/nginx

Li Chi in the early morning of September 19, 2021

Posted by subkida on Mon, 20 Sep 2021 11:44:38 -0700

Programmer Group

Golang practice record: using gin framework to realize forwarding function: forwarding using nginx

summary

Back end service

nginx container startup and restart

Some tests of nginx forwarding

experimental result

Summary

reference resources

summary

Back end service

nginx container startup and restart

Some tests of nginx forwarding

experimental result

Summary

reference resources

Hot Keywords