SpringBoot integration FastDFS+Nginx integration Token-based anti-theft chain

Why use SpringBoot?

SpringBoot is a new framework provided by the Pivotal team designed to simplify the initial build and development of new Spring applications.The framework is configured in a specific way so that developers no longer need to define a template configuration.In this way, Spring Boot is committed to becoming a leader in the burgeoning field of rapid application development.

Create a stand-alone Spring application
Embedded Tomcat without deploying WAR files
Simplify Maven configuration
Automatically configure Spring
Provide production-ready functions such as metrics, health checks, and external configuration
Absolutely no code generation and no configuration requirements for XML

Why use Nginx?

Summary

Nginx (engine x) is an open source software that supports highly concurrent WW services and proxy services.Nginx was developed by Russian Igor Sysoev and was originally used on large Russian websites (www.rambler.ru).Later, the author opened the source code as a BSD-like license for global use.In terms of functional applications, Nginx is not only an excellent Web service software, but also has the functions of reverse proxy load balancing and caching.In terms of reverse proxy load balancing, it is similar to your professional proxy software such as LVS load balancing and HAProxy.Nginx is easier to deploy and has professional cache service software like Squid in terms of cache service functionality.Nginx can run on UNIX, Linux, MS Windows Server, Mac OS X Server, Solaris and other operating systems.
Important features of Nginx
1. It can be accessed and cached concurrently for static resource high-speed nodes.
2. You can use reverse proxy acceleration and you can cache data.
3. It has simple load balancing, node health check and fault tolerance capabilities.
4. Supports cache acceleration for remote Fast CGI services.
5. Supports Fast CGI, Uwsgi, SCGI, Memcached Server acceleration and caching.
6. Supports SSL, TLS, SNI.
7. Has a modular architecture.
8. Filters include gzip compression, ranges support, chunked response, XSLT, SSL, and image scaling.
9. In an SL filter, there are multiple SSL pages that can be processed in parallel via Fast CGI or reverse proxy.
WWW Service Features of Nginx
1. Supports domain name, port, and IP-based virtual host configuration.
2. KeepAlived and piplined connections are supported.
3. It can be configured and managed simply, conveniently and flexibly.
4. Support for modifying Nginx configuration and smooth restart without interrupting business access when code is online.
5. Customizable access log format, temporary buffering for log write operations, fast log polling, and processing logs through rsyslog.
6. Signals can be used to control the Nginx process.
7. Supports 3xx-5xxHTTP status code redirection.
8. Supports rewrite modules, URI rewriting and regular expression matching.
9. Support access control based on client IP address and HTTP basic authentication.
10. Supports special HTTP request methods such as PUT, DELETE, MKCOL, COPY, MOVE, etc.
11. Supports FLV streaming and MP4 streaming technology product applications.
12. Supports HTTP response rate limits.
13. Supports concurrent connections or request restrictions for the same IP address.
14. Support mail service proxy.
15. Supports high concurrency and can support millions of concurrent connections.
16. Low resource consumption, threads that can open 10 nginx under 30,000 concurrent connections consume less than 200 MB of memory.
17. Can do HTTP reverse proxy and accelerated caching, and load balancing functions, built-in health check for RS node servers, discounted but with professional HAProxy or LVS functions.
18. It has the caching function of professional caching software such as Squid.
19. Supports the asynchronous network I/O event model epoll (Linux 2.6+).
Nginx Software Major Enterprise Applications
1. As Web service software.
2. Use Nginx to run static data such as HTML, JS, CSS, small pictures (similar to Lighttpd).
3. Run dynamic programs such as PHP with Fast CGI (for example, using fastcgi_pass).
4. Nginx combines Tomcat/Resin and other support for Java dynamic programs (commonly proxy_pass).
5. Reverse proxy or load balancing service (Nginx has supported TCP proxy since 1.9.0).
6. Front-end business data caching service.
Performance comparison of Web service application products
1. On the access of static data: when processing small files (less than 1MB), Nginx and Lighttpd have advantages over Apache, Nginx has obvious advantages in handling small files, and Lighttpd has the strongest synthesis.
2. Dynamic data access: Apache has an advantage because the ability to process dynamic data lies in the service capabilities of PHP (Java) and back-end databases, that is, the bottleneck is not on the Web server.
3. In general, the concurrent connection reference values supported by the normal PHP engine are 300~1000.Reference values for concurrent connections between the Java engine and the database range from 300 to 1500.
Why does Nginx perform better than Apache?
1. Nginx uses the latest version of eepoll (Linux 2.6 kernel) and kqueue (FreeBSD) asynchronous network I/O models, while Apache uses the traditional select model.
2. Currently, the Squid and Memcached software which can withstand high concurrent access under Linux are all epoll models.
3. Apache uses a lower select network I/O model when dealing with large numbers of connections.
How to use Web server correctly?
1. Static business: If you have a high concurrency scenario, try using Nginx or Lighttpd, with Nginx preferred.
2. Dynamic Business: In theory, both Nginx and Apache can be used. It is recommended to use Nginx. To avoid the software diversification of the same business services and increase maintenance costs, dynamic business can use Nginx as front-end proxy, and then forward to other servers according to the elements or directories of the page for processing.
3. Use Nginx for both dynamic and static business.

Deployment is no longer a duplicate, but if you need to, move to FastDFS Distributed File Cluster for Java Advanced Architecture: https://blog.51cto.com/xvjunjie/2377669

Create a project using the IDEA scene launcher

Create a Maven project, modify the POM.xml file to add the following dependencies:

<dependencies>
    <!-- SpringBoot Auto-configure dependencies -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-autoconfigure</artifactId>
        <version>1.5.20.RELEASE</version>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-configuration-processor</artifactId>
        <version>1.5.20.RELEASE</version>
    </dependency>
    <!-- Log-related dependencies -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-logging</artifactId>
        <version>1.5.20.RELEASE</version>
    </dependency>
    <!-- Object pool-related dependencies -->
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-pool2</artifactId>
        <version>2.6.0</version>
    </dependency>

</dependencies>

Create necessary packages

annotation: store relevant notes
autoconfiguation: Stores autoconfigure classes
factory:Store factory Class
properties:Stores the configuration parameter class
service:Store service Class

In general, SpringBoot provides the corresponding @EnableXxx annotation to turn on a function on the main boot class of the application:

// EnableFastdfsClient.java
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Import(FastdfsAutoConfiguration.class)
@Documented
public @interface EnableFastdfsClient {
}

The following are the related auto-configuration classes:

// FastdfsAutoConfiguration.java
@Configuration
@EnableConfigurationProperties(FastdfsProperties.class)
public class FastdfsAutoConfiguration {
    @Autowired
    private FastdfsProperties fastdfsProperties;

    @Bean
    @ConditionalOnMissingBean(FastdfsClientService.class)
    public FastdfsClientService fastdfsClientService() throws Exception {
        return new FastdfsClientService(fastdfsProperties);
    }
}

Create related factory classes:

// StorageClientFactory.java
// Factory class for creating connection objects
public class StorageClientFactory implements PooledObjectFactory<StorageClient> {

    @Override
    public PooledObject<StorageClient> makeObject() throws Exception {
        TrackerClient client = new TrackerClient();
        TrackerServer server = client.getConnection();
        return new DefaultPooledObject<>(new StorageClient(server, null));
    }

    @Override
    public void destroyObject(PooledObject<StorageClient> p) throws Exception {
        p.getObject().getTrackerServer().close();
    }

    @Override
    public boolean validateObject(PooledObject<StorageClient> p) {
        return false;
    }

    @Override
    public void activateObject(PooledObject<StorageClient> p) throws Exception {

    }

    @Override
    public void passivateObject(PooledObject<StorageClient> p) throws Exception {

    }
}

The Properties class is used to map the application.properties or application.yml configuration file:

// FastdfsProperties.java
@ConfigurationProperties(prefix = "fastdfs")
public class FastdfsProperties {
    // Connection timeout
    // Network Timeout
    // Character Set Encoding
    // Whether to use Token
    // Token Encryption Key
    // Tracker IP address, separated by semicolons
    // Maximum number of connection objects in connection pool
    // Maximum number of free objects in connection pool
    // Minimum number of free objects in connection pool
    // Nginx server IP, multiple using semicolons
    // The tolerable wait time (milliseconds) to get the connected object
    private String connectTimeout = "5";
    private String networkTimeout = "30";
    private String charset = "UTF-8";
    private String httpAntiStealToken = "false";
    private String httpSecretKey = "";
    private String httpTrackerHttpPort = "";
    private String trackerServers = "";
    private String connectionPoolMaxTotal = "18";
    private String connectionPoolMaxIdle = "18";
    private String connectionPoolMinIdle = "2";
    private String nginxServers = "";

    // Related Setter and Getter methods need to be created
}

To encapsulate methods in the Service class, only three commonly used methods are shown below:

// FastdfsClientSerivce.java
public class FastdfsClientService {
    // SpringBoot Loaded Configuration File
    // Connection pool configuration item
    // Converted Configuration Entry
    // Connection Pool
    // Nginx Server Address
    private FastdfsProperties fdfsProp;
    private GenericObjectPoolConfig config;
    private Properties prop;
    private GenericObjectPool<StorageClient> pool;
    private String[] nginxServers;
    private Logger logger;

    public FastdfsClientService(FastdfsProperties fdfsProp) throws Exception {
        this.fdfsProp = fdfsProp;
        this.logger = LoggerFactory.getLogger(getClass());
        init();
        create();
        info();
    }

    /**
     * Initialize Global Client
     */
    private void init() throws Exception {
        this.prop = new Properties();
        this.logger.info("FastDFS: reading config file...");
        this.logger.info("FastDFS: fastdfs.connect_timeout_in_seconds=" + this.fdfsProp.getConnectTimeout());
        this.logger.info("FastDFS: fastdfs.network_timeout_in_seconds=" + this.fdfsProp.getNetworkTimeout());
        this.logger.info("FastDFS: fastdfs.charset=" + this.fdfsProp.getCharset());
        this.logger.info("FastDFS: fastdfs.http_anti_steal_token=" + this.fdfsProp.getHttpAntiStealToken());
        this.logger.info("FastDFS: fastdfs.http_secret_key=" + this.fdfsProp.getHttpSecretKey());
        this.logger.info("FastDFS: fastdfs.http_tracker_http_port=" + this.fdfsProp.getHttpTrackerHttpPort());
        this.logger.info("FastDFS: fastdfs.tracker_servers=" + this.fdfsProp.getTrackerServers());
        this.logger.info("FastDFS: fastdfs.connection_pool_max_total=" + this.fdfsProp.getConnectionPoolMaxTotal());
        this.logger.info("FastDFS: fastdfs.connection_pool_max_idle=" + this.fdfsProp.getConnectionPoolMaxIdle());
        this.logger.info("FastDFS: fastdfs.connection_pool_min_idle=" + this.fdfsProp.getConnectionPoolMinIdle());
        this.logger.info("FastDFS: fastdfs.nginx_servers=" + this.fdfsProp.getNginxServers());

        this.prop.put("fastdfs.connect_timeout_in_seconds", this.fdfsProp.getConnectTimeout());
        this.prop.put("fastdfs.network_timeout_in_seconds", this.fdfsProp.getNetworkTimeout());
        this.prop.put("fastdfs.charset", this.fdfsProp.getCharset());
        this.prop.put("fastdfs.http_anti_steal_token", this.fdfsProp.getHttpAntiStealToken());
        this.prop.put("fastdfs.http_secret_key", this.fdfsProp.getHttpSecretKey());
        this.prop.put("fastdfs.http_tracker_http_port", this.fdfsProp.getHttpTrackerHttpPort());
        this.prop.put("fastdfs.tracker_servers", this.fdfsProp.getTrackerServers());
        ClientGlobal.initByProperties(this.prop);
    }
    /**
     * Display initialization information
     */
    private void info() {
        this.logger.info("FastDFS parameter: ConnectionPoolMaxTotal ==> " + this.pool.getMaxTotal());
        this.logger.info("FastDFS parameter: ConnectionPoolMaxIdle ==> " + this.pool.getMaxIdle());
        this.logger.info("FastDFS parameter: ConnectionPoolMinIdle ==> " + this.pool.getMinIdle());
        this.logger.info("FastDFS parameter: NginxServer ==> " + Arrays.toString(this.nginxServers));
        this.logger.info(ClientGlobal.configInfo());
    }

    /**
     * Create connection pool
     */
    private void create() {
        this.config = new GenericObjectPoolConfig();
        this.logger.info("FastDFS Client: Creating connection pool...");
        this.config.setMaxTotal(Integer.parseInt(this.fdfsProp.getConnectionPoolMaxTotal()));
        this.config.setMaxIdle(Integer.parseInt(this.fdfsProp.getConnectionPoolMaxIdle()));
        this.config.setMinIdle(Integer.parseInt(this.fdfsProp.getConnectionPoolMinIdle()));
        StorageClientFactory factory = new StorageClientFactory();
        this.pool = new GenericObjectPool<StorageClient>(factory, this.config);
        this.nginxServers = this.fdfsProp.getNginxServers().split(",");
    }

    /**
     * Nginx Server Load Balancing Algorithm
     *
     * @param servers server address
     * @param address Client IP Address
     * @return Available server addresses
     */
    private String getNginxServer(String[] servers, String address) {
        int size = servers.length;
        int i = address.hashCode();
        int index = abs(i % size);
        return servers[index];
    }

    /**
     * Download with anti-theft chain
     *
     * @param fileGroup       File Group Name
     * @param remoteFileName  Remote File Name
     * @param clientIpAddress Client IP Address
     * @return Full URL Address
     */
    public String autoDownloadWithToken(String fileGroup, String remoteFileName, String clientIpAddress) throws Exception {
        int ts = (int) (System.currentTimeMillis() / 1000);
        String token = ProtoCommon.getToken(remoteFileName, ts, ClientGlobal.getG_secret_key());
        String nginx = this.getNginxServer(this.nginxServers, clientIpAddress);
        return "http://" + nginx + "/" + fileGroup + "/" + remoteFileName + "?token=" + token + "&ts=" + ts;
    }

    /**
     * Upload files for uploading pictures
     *
     * @param buffer Byte Array
     * @param ext    Extension
     * @return File group name and ID
     */
    public String[] autoUpload(byte[] buffer, String ext) throws Exception {
        String[] upload = this.upload(buffer, ext, null);
        return upload;
    }

    /**
     * Download without anti-theft chain, unlocking anti-theft chain will cause the method to throw an exception
     *
     * @param fileGroup       File Group Name
     * @param remoteFileName  Remote File ID
     * @param clientIpAddress Client IP address to assign Nginx server based on client IP
     * @return Full URL Address
     */
    public String autoDownloadWithoutToken(String fileGroup, String remoteFileName, String clientIpAddress) throws Exception {
        if (ClientGlobal.getG_anti_steal_token()) {
            this.logger.error("FastDFS Client: You've turned on Token authentication.");
            throw new Exception("You've turned on Token authentication.");
        }
        String nginx = this.getNginxServer(this.nginxServers, clientIpAddress);
        return "http://" + nginx + fileGroup + "/" + remoteFileName;
    }

    // There are still many ways to do this, so we will not show them all.
}

To use the convenient configuration prompt function in IDEA, we need to create a metadata file (resources/spring-configuration-metadata.json):

{
  "groups": [
    {
      "name": "fastdfs",
      "type": "com.bluemiaomiao.properties.FastdfsProperties",
      "sourceType": "com.bluemiaomiao.properties.FastdfsProperties"
    }
  ],
  "properties": [
    {
      "name": "connectTimeout",
      "type": "java.lang.String",
      "sourceType": "com.bluemiaomiao.properties.FastdfsProperties",
      "defaultValue": "5"
    },
    {
      "name": "networkTimeout",
      "type": "java.lang.String",
      "sourceType": "com.bluemiaomiao.properties.FastdfsProperties",
      "defaultValue": "30"
    },
    {
      "name": "charset",
      "type": "java.lang.String",
      "defaultValue": "UTF-8"
    },
    {
      "name": "httpAntiStealToken",
      "type": "java.lang.String",
      "sourceType": "com.bluemiaomiao.properties.FastdfsProperties",
      "defaultValue": "false"
    },
    {
      "name": "httpSecretKey",
      "type": "java.lang.String",
      "sourceType": "com.bluemiaomiao.properties.FastdfsProperties"
    },
    {
      "name": "httpTrackerHttpPort",
      "type": "java.lang.Integer",
      "sourceType": "com.bluemiaomiao.properties.FastdfsProperties"
    },
    {
      "name": "trackerServers",
      "type": "java.lang.String",
      "sourceType": "com.bluemiaomiao.properties.FastdfsProperties"
    },
    {
      "name": "connectionPoolMaxTotal",
      "type": "java.lang.Integer",
      "sourceType": "com.bluemiaomiao.properties.FastdfsProperties",
      "defaultValue": "18"
    },
    {
      "name": "connectionPoolMaxIdle",
      "type": "java.lang.Integer",
      "sourceType": "com.bluemiaomiao.properties.FastdfsProperties",
      "defaultValue": "18"
    },
    {
      "name": "connectionPoolMinIdle",
      "type": "java.lang.Integer",
      "sourceType": "com.bluemiaomiao.properties.FastdfsProperties",
      "defaultValue": "2"
    },
    {
      "name": "nginxServers",
      "type": "java.lang.String",
      "sourceType": "com.bluemiaomiao.properties.FastdfsProperties"
    }
  ],
  "hints": [
    {
      "name": "http_anti_steal_token",
      "values": [
        {
          "value": "false"
        },
        {
          "value": "true"
        }
      ]
    }
  ]
}

Add a custom starter to the project

Create a SpringBoot project, check the Web option, version 1.5.20
Execute mvn clean install to install it locally into the project directory of the scene launcher
Add dependencies to the POM.xml file:

<dependency>
    <groupId>com.bluemiaomiao</groupId>
    <artifactId>fastdfs-spring-boot-starter</artifactId>
    <version>1.0-SNAPSHOT</version>
</dependency>

Remember to turn on automatic import of IDEA

Create configuration file application.properties

fastdfs.nginx-servers=192.168.80.2:8000,192.168.80.3:8000,192.168.80.4:8000
fastdfs.tracker-servers=192.168.80.2:22122,192.168.80.3:22122,192.168.80.4:22122
fastdfs.http-secret-key=2scPwMPctXhbLVOYB0jyuyQzytOofmFCBIYe65n56PPYVWrntxzLIDbPdvDDLJM8QHhKxSGWTcr+9VdG3yptkw
fastdfs.http-anti-steal-token=true
fastdfs.http-tracker-http-port=8080
fastdfs.network-timeout=30
fastdfs.connect-timeout=5
fastdfs.connection-pool-max-idle=18
fastdfs.connection-pool-min-idle=2
fastdfs.connection-pool-max-total=18
fastdfs.charset=UTF-8

Or use application.yml

fastdfs:
  charset: UTF-8
  connect-timeout: 5
  http-secret-key: 2scPwMPctXhbLVOYB0jyuyQzytOofmFCBIYe65n56PPYVWrntxzLIDbPdvDDLJM8QHhKxSGWTcr+9VdG3yptkw
  network-timeout: 30
  http-anti-steal-token: true
  http-tracker-http-port: 8080
  connection-pool-max-idle: 20
  connection-pool-max-total: 20
  connection-pool-min-idle: 2
  nginx-servers: 192.168.80.2:8000,192.168.80.3:8000,192.168.80.4:8000
  tracker-servers: 192.168.80.2:22122,192.168.80.3:22122,192.168.80.4:22122

Create Controller Class Test Method

// controllers.DownloadController.java
@Controller
@RequestMapping(value = "/download")
public class DownloadController {

    @Autowired
    private FastdfsClientService service;

    @ResponseBody
    @RequestMapping(value = "/image")
    public String image() throws Exception {
        // Previously uploaded data, practical scenarios should use SQL databases to store
        return service.autoDownloadWithToken("group1", "M00/00/00/wKhQA1ysjSGAPjXbAAVFOL7FJU4.tar.gz", "192.168.80.1");
    }
}

Project Home Page: https://github.com/bluemiaomiao/fastdfs-spring-boot-starter
Home page for domestic projects: https://gitee.com/bluemiaomiao/fastdfs-spring-boot-starter

Posted by scorpioy on Fri, 10 May 2019 19:46:11 -0700

Programmer Group