Universal File Service Component (Netty Implementation Version)

Keywords: Java Netty Spring Tomcat socket

The file service components described in this article have been described in the previous article of the author.( File upload and download component based on netty However, this paper will be based on the previous implementation to upgrade again, using annotation-based way of automatic assembly.

1. Introduction

1.1 Introduction to Netty

Netty is an asynchronous event-driven network application framework for rapid development of maintainable high-performance protocol servers and clients. Detailed introduction can be referred to. Netty official website.

Netty is a NIO client server framework which enables quick and easy development of network applications such as protocol servers and clients. It greatly simplifies and streamlines network programming such as TCP and UDP socket server.

Netty's main features:

Unified API for various transport types - blocking and non-blocking socket (Unified API)
Based on a flexible and extensible event model which allows clear separation of concerns (event model)
Highly customizable thread model - single thread, one or more thread pools such as SEDA (thread model)
True connection less datagram Socket support (since 3.1) (connectionless datagram Socket support)

Introduction to 1.2 Component Functions

This component is based on netty 3.6.3. It has the following functions: file upload, file replacement, file deletion, if it is a picture, it can also generate thumbnails and other functions. Simple to use, only need to introduce commons-doc-client-netty, can achieve the above operation of the file.

This component is divided into three module s, namely:

commons-doc-server-netty: Netty implements the server side of file service component
commons-doc-common: Netty File Service Component Common Component
commons-doc-client-http: Client of Netty File Service Component

2. Server

2.1 Function Brief Introduction

Server-side components achieve the following functions: file upload, file replacement, file deletion, if it is a picture, it can also generate thumbnails and other functions. The code structure is shown in the following figure.

All file services are based on the interface DocServer Processor. There are mainly the following implementation classes:

Upload Doc Server Handler Implements File Upload Service
Replace Doc Server Handler Implementing File Replacement Service
DeleteDoc Server Handler Implementing File Delete Service
Create Thumb Picture Server Handler Implements Create Picture Thumb Service

2.2 Implementation steps

Specific implementation steps take file upload as an example.

First, the org.fortune.doc.server.support.DocServerHandler class continuously listens to client requests, and if it is a file processing action, it enters the messageReceived method for the corresponding processing logic. This class defines the following member variables:

//http request
private HttpRequest request;
//Do you need to continue the work at breakpoint?
private boolean readingChunks;
//Received file content
private final StringBuffer responseContent = new StringBuffer();
//Resolve the files received
private static final HttpDataFactory factory = new DefaultHttpDataFactory(DefaultHttpDataFactory.MINSIZE); //16384L
//The decoding class for post requests, which decodes bytes into Http requests.
private HttpPostRequestDecoder decoder;
//Request parameters
private RequestParam requestParams = new RequestParam();

In the implementation of this method, if the file size is less than the minimum value of chunked, the file upload operation is carried out directly. Otherwise, it needs to be partitioned. Then upload the file.
Operation with file size less than 1k:

if (request.isChunked()) { //Explain that the request has not been completed, continue
    this.readingChunks = true;
    LOGGER.info("File Blocking....");
} else {
    LOGGER.info("File size less than 1 KB，File receiving is completed, and the corresponding file processing operation is carried out directly.....");
    //When the request is completed, the request parameters are received and initialized.
    RequestParamParser.parseParams(this.decoder, this.requestParams);
    //Document operation according to request parameters
    LOGGER.info("File Processing Begins....requestParams Parametric analysis:{}",requestParams);
    String result = DocServerHandlerFactory.process(this.requestParams);
    LOGGER.info("End of File Processing....FileServerHandlerFactory Processing results:{}",result);
    this.responseContent.append(result);
    //Response information to client
    writeResponse(e.getChannel());

    e.getFuture().addListener(ChannelFutureListener.CLOSE);
}

Block processing operations are required:

HttpChunk chunk = (HttpChunk) e.getMessage();
try {
    //chunk.getContent().capacity();
    LOGGER.info("File Blocking....File size:{} bytes",chunk.getContent().capacity());
    this.decoder.offer(chunk);
} catch (HttpPostRequestDecoder.ErrorDataDecoderException e1) {
    e1.printStackTrace();
    this.responseContent.append(e1.getMessage());
    writeResponse(e.getChannel());
    Channels.close(e.getChannel());
    return;
}

if (chunk.isLast()) {
    //end of file
    this.readingChunks = false;
    LOGGER.info("Arrive at the end of the file content, and do the corresponding file processing operation.....start");
    RequestParamParser.parseParams(this.decoder, this.requestParams);

    LOGGER.info("File Processing Begins....requestParams Parametric analysis:{}",requestParams);
    String result = DocServerHandlerFactory.process(this.requestParams);
    LOGGER.info("End of File Processing....FileServerHandlerFactory Processing results:{}",result);

    this.responseContent.append(result);
    //Response information to client
    writeResponse(e.getChannel());

    e.getFuture().addListener(ChannelFutureListener.CLOSE);
    LOGGER.info("Arrive at the end of the file content, and do the corresponding file processing operation.....end");
}

There are two main points for attention in the above operations:

Parsing of request parameters (assignment of parameters based on HttpDataType)
According to the parsed parameters, the corresponding file processing operations (according to the type of file operation, select the appropriate processing handle for file processing)

3. Client

3.1 Function Brief Introduction

The client component mainly provides the interface for accessing the server component. It provides the following interfaces: file upload, file replacement, file deletion, and if it is a picture, it can also generate thumbnails and other functions. The code structure is as follows:

The org.fortune.doc.client.DocClient class is a tool class that provides interfaces to the outside world. It has the following main methods:

uploadFile file upload, the corresponding file processing handle class is: org. fortune. doc. client. handler. Upload DocClientHandler
deleteFile deletes server-side files, and the corresponding file processing handle class is: org.fortune.doc.client.handler.DeleteDocClientHandler.
ReplceFile replaces server-side files with the corresponding file processing handle class: org.fortune.doc.client.handler.ReplaceDocClientHandler
createThumbPicture generates thumbnails corresponding to the file processing handle class: org.fortune.doc.client.handler.CreateThumbPictureClientHandler

3.2 Implementation steps

The implementation steps take uploading files as an example, and other similar implementations. Code directly:

/**
     * File upload
     * @param file Files to be uploaded
     * @param fileName File name
     * @param thumbMark Need to generate thumbnails
     * @return
     * @author:landyChris
     */
    public static String uploadFile(File file, String fileName,
                                    boolean thumbMark) {
        DocClientPipelineFactory clientPipelineFactory = new DocClientPipelineFactory();
        //Auxiliary classes. Used to help us create NETTY services
        ClientBootstrap bootstrap = createClientBootstrap(clientPipelineFactory);
        String strThumbMark = Constants.THUMB_MARK_NO;
        if (thumbMark) {
            strThumbMark = Constants.THUMB_MARK_YES;
        }
        //Processing Upload File Logic Specifically
        uploadFile(bootstrap, DocClientContainer.getInstance().getHost(),
                DocClientContainer.getInstance().getPort(), file, fileName, strThumbMark,
                DocClientContainer.getInstance().getUserName(),
                DocClientContainer.getInstance().getPassword());
        Result result = clientPipelineFactory.getResult();
        if ((result != null) && (result.isSuccess())) {
            return result.getFilePath();
        }
        return null;
    }

With three parameters, the first few lines of code are a lot of netty initialization work, specifically look at a private method uploadFile, as shown in the following code:

private static void uploadFile(ClientBootstrap bootstrap, String host,
                                   int port, File file, String fileName, String thumbMark,
                                   String userName, String pwd) {
        //1. Building uri objects
        URI uri = getUri(host, port);
        //2. Connect the netty server
        ChannelFuture future = bootstrap.connect(new InetSocketAddress(host,
                port));
        //3. Asynchronous acquisition of Channel objects
        Channel channel = future.awaitUninterruptibly().getChannel();
        if (!future.isSuccess()) {
            future.getCause().printStackTrace();
            bootstrap.releaseExternalResources();
            return;
        }
        //4. Initialize file upload handle object
        AbstractDocClientHandler handler = new UploadDocClientHandler(host, uri,
                file, fileName, thumbMark, userName, pwd);
        //5. Get the Request object
        HttpRequest request = handler.getRequest();
        //6. Obtain Http Data Processing Factory
        HttpDataFactory factory = getHttpDataFactory();
        //7. Packing the data, mainly setting the parameters needed to upload the file. At this time, the handle is the specific UploadFileClientHandler object.
        HttpPostRequestEncoder bodyRequestEncoder = handler
                .wrapRequestData(factory);
        //8. Write the request into the pipeline and transmit it to the server
        channel.write(request);
        //9. Do some actions to close resources
        if (bodyRequestEncoder.isChunked()) {
            channel.write(bodyRequestEncoder).awaitUninterruptibly();
        }
        bodyRequestEncoder.cleanFiles();
        channel.getCloseFuture().awaitUninterruptibly();

        bootstrap.releaseExternalResources();
        factory.cleanAllHttpDatas();
    }

The main steps are as follows:

Building uri objects
Connect to netty server
Asynchronous acquisition of Channel objects
Initialize file upload handle object
Get the Request object
Getting Http Data Processing Factory
Packing of data is mainly to set the parameters needed to upload files. At this time, the handle invoked is the specific UploadFileClientHandler object.
Write the request in the pipeline and transmit it to the server
Do some actions to close resources
Refer to the code on github for details. If you like it, you can add a star ha.

4. Operational Guidelines

The use of the file service component needs to be divided into two parts, one is the server configuration and start-up, and the other is the client configuration and start-up.

4.1 Server Configuration and Startup

4.1.1 Configuration

The configuration of server side adopts the configuration of yml file, which is more concise and concise. The main point of attention is the configuration of file storage location. In the development process, there are two ways to configure:

Idea self-startup mode: If this mode is adopted, you need to configure rootPath under the project path (target directory), as follows:

# To execute in idea, you need to configure the packaged files in the target directory
    rootPath: C:\03_code\idea_workspace\fortune-commons\commons-doc-server-netty\target\commons-doc-server-netty\ #Upload the root directory of the file, the actual working environment can be changed according to the actual situation.

Independent start-up mode of tomcat after packaging

# You can also copy the packaged war package to tomcat webapp directory and run it directly.
    rootPath: C:\05_webserver\apache-tomcat-8.5.42\webapps\doc-server #Upload the root directory of the file, the actual working environment can be changed according to the actual situation.

4.1.2 Start-up

This article uses idea self-startup mode, then need to configure the tomcat path and introduce the corresponding module, as shown in the following figure:

When the configuration is complete, the file service component can be started. The following figure is the startup information log:

....
2019-07-20  23:48:56.174 [RMI TCP Connection(3)-127.0.0.1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Creating shared instance of singleton bean 'mvcViewResolver'
2019-07-20  23:48:56.182 [RMI TCP Connection(3)-127.0.0.1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Creating shared instance of singleton bean 'viewResolver'
2019-07-20  23:48:56.208 [RMI TCP Connection(3)-127.0.0.1] INFO  org.fortune.commons.core.help.BeanInitializeCompletedListener - spring conf Container initialization completed..Handling post-startup events--start
2019-07-20  23:48:56.212 [RMI TCP Connection(3)-127.0.0.1] INFO  org.fortune.doc.server.DocServerContainer - Join the account: fortune
2019-07-20  23:48:56.212 [RMI TCP Connection(3)-127.0.0.1] INFO  org.fortune.doc.server.DocServerContainer - Join the account: fortune0
2019-07-20  23:48:56.213 [RMI TCP Connection(3)-127.0.0.1] INFO  org.fortune.doc.server.DocServerContainer - Add default account: Account{userName='default_account', password='lyx', rootPath='C:\05_webserver\apache-tomcat-8.5.42\bin\', level=1, thumbHeight=20, thumbWidth=20}
2019-07-20  23:48:56.296 [RMI TCP Connection(3)-127.0.0.1] INFO  org.fortune.commons.core.help.BeanInitializeCompletedListener - spring conf Container initialization completed..Handling post-startup events--end
2019-07-20  23:48:56.302 [RMI TCP Connection(3)-127.0.0.1] DEBUG org.springframework.jndi.JndiTemplate - Looking up JNDI object with name [java:comp/env/spring.liveBeansView.mbeanDomain]
2019-07-20  23:48:56.303 [RMI TCP Connection(3)-127.0.0.1] DEBUG org.springframework.jndi.JndiLocatorDelegate - Converted JNDI name [java:comp/env/spring.liveBeansView.mbeanDomain] not found - trying original name [spring.liveBeansView.mbeanDomain]. javax.naming.NameNotFoundException: Name [spring.liveBeansView.mbeanDomain] is not bound in this Context. Unable to find [spring.liveBeansView.mbeanDomain].
2019-07-20  23:48:56.303 [RMI TCP Connection(3)-127.0.0.1] DEBUG org.springframework.jndi.JndiTemplate - Looking up JNDI object with name [spring.liveBeansView.mbeanDomain]
2019-07-20  23:48:56.304 [RMI TCP Connection(3)-127.0.0.1] DEBUG org.springframework.jndi.JndiPropertySource - JNDI lookup for name [spring.liveBeansView.mbeanDomain] threw NamingException with message: Name [spring.liveBeansView.mbeanDomain] is not bound in this Context. Unable to find [spring.liveBeansView.mbeanDomain].. Returning null.
2019-07-20  23:48:56.307 [RMI TCP Connection(3)-127.0.0.1] INFO  org.springframework.web.context.ContextLoader - Root WebApplicationContext initialized in 2216 ms
2019-07-20  23:48:56.311 [RMI TCP Connection(3)-127.0.0.1] DEBUG org.springframework.web.filter.CharacterEncodingFilter - Filter 'characterEncodingFilter' configured for use
2019-07-20  23:48:56.320 [RMI TCP Connection(3)-127.0.0.1] INFO  org.springframework.web.servlet.DispatcherServlet - Initializing Servlet 'doc-server'
....

Then you can access the following page, which shows as follows:

But the netty port we configured is 9999. Let's try to access it.

At this point, we can directly upload files on the web side without relying on the client side of the test operation. Enter the account, password, select the appropriate file to upload, upload the successful return page as follows:

The log information printed by the background console is as follows:

2019-07-20  23:52:12.833 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler-File Blocking Operation...
2019-07-20  23:52:12.835 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler-File Blocking Operation.... File Size: 1024 bytes
2019-07-20  23:52:12.849 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler-File Blocking Operation.... File Size: 2048 bytes
2019-07-20  23:52:12.849 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler-File Blocking Operation.... File Size: 2048 bytes
2019-07-20  23:52:12.850 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler-File Blocking Operation.... File Size: 3072 bytes
2019-07-20  23:52:12.850 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler - File block operation.... File size: 4096 bytes
2019-07-20  23:52:12.850 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler-File Blocking Operation.... File Size: 6144 bytes
2019-07-20  23:52:12.852 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler - File block operation.... File size: 8192 bytes
2019-07-20  23:52:12.852 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler - File block operation.... File size: 8192 bytes
2019-07-20  23:52:12.852 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler-File Blocking Operation.... File Size: 3072 bytes
2019-07-20  23:52:12.853 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler - File block operation.... File size: 8192 bytes
2019-07-20  23:52:12.853 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler-File Blocking Operation.... File Size: 7168 bytes
2019-07-20  23:52:12.853 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler - File block operation.... File size: 8192 bytes
2019-07-20  23:52:12.854 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler - File block operation.... File size: 8192 bytes
2019-07-20  23:52:12.854 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler-File Blocking Operation.... File Size: 6144 bytes
2019-07-20  23:52:12.856 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler - File block operation.... File size: 8157 bytes
2019-07-20  23:52:12.862 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler-File Blocking Operation.... File Size: 0 bytes
2019-07-20  23:52:12.862 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler-arrives at the end of the file content and performs the corresponding file processing operations...
2019-07-20  23:52:12.865 [New I/O worker #1] INFO org. fortune. doc. server. support. DocServerHandler - File processing begins.... RequParams parameter parsing:
NETTY WEB Server
===================================


UserName=fortune
pwd=fortune
action=uploadFile
fileContentType=image/jpeg
fileSize=81 KB 
getform=POST
Send=Send

2019-07-20  23:52:12.867 [New I/O worker #1] INFO org.fortune.doc.server.handler.factory.DocServerHandlerFactory-File upload operation...
2019-07-20  23:52:12.869 [New I/O worker #1] INFO  org.fortune.doc.server.handler.UploadDocServerHandler - --srcFileName--psb.jpg
2019-07-20  23:52:12.871 [New I/O worker #1] INFO org.fortune.doc.server.handler.UploadDocServerHandler-file upload success, save the path: fortune\ l\\\\\\\\\\\\\\\\\\\0235212_1027.jpg
2019-07-20  23:52:12.871 [New I/O worker #1] DEBUG org. fortune. doc. server. handler. Upload Doc Server Handler - Generate thumbnails
2019-07-20  23:52:12.872 [New I/O worker #1] INFO org. fortune. doc. server. handler. UploadDocServerHandler - Generating thumbnail name: 1900232212_1027_thumb. jpg, path: C:\03_code\\\idea_workspacefortune-commons\ doc-server-server-netty\\\\\\\\\\\jpg
2019-07-20  23:52:13.130 [New I/O worker #1) DEBUG org.fortune.doc.server.handler.factory.DocServerHandler Factory-Execution result: {"action": "uploadFile", "code": 1, "filePath", "fortune\\\\\\2_1027.jpg", "msg", "successful file upload", "true}
2019-07-20  23:52:13.130 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler-File processing end....FileServerHandler Factory processing results: {"action": "uploadFile", "code": 1, "filePath": "fortune\\\\\\\\\\\\\
2019-07-20  23:52:13.133 [New I/O worker #1] INFO org.fortune.doc.server.support.DocServerHandler-arrives at the end of the file content, and performs the corresponding file processing operations...

Look again at the target directory in the netty code, where there are uploaded files, as follows:

We can access it at the following address.

http://localhost:8080/doc-server/fortune//l//190720235212_1027.jpg. For how to get the picture address, there are corresponding methods in the client tool class.

4.2 Client Configuration and Startup

4.2.1 Configuration

The configuration of client is relatively simple, and it is also configured in the form of yml file as follows:

#After successful upload, configure the following server address in the web project which needs to access the file. The correct format is as follows:
# ${host}/${port}/${appName}/${path}
# appName is the name of the remote file service app, such as doc-server
# Path is the relative path of the file returned by the server after uploading the file
#http://localhost:8080/doc-server/fortune//p//190629082821_8300.jpg
upload:
  server:
    port: 9999 #What you need to configure is the remote file server netty service port number
    host: 127.0.0.1
    access: #access files
      port: 8080 #Access file web service port number
      domainName: doc-server #Access file web service application name
  userName: fortune
  password: fortune

It should be noted that the port number of file access is different from the port corresponding to the remote file server netty, which requires special attention. When calling the path returned by the file service, the address of the server to access the file is needed, and then the corresponding file content is accessed. The corresponding server address can be obtained by the method org.fortune.doc.client.DocClientContainer#getDocServerUrl, and the complete address of the file can be obtained by splicing the relative path returned.

4.2.2 Start/Call

The client unit test uses, for example, the following:

/**
 * @author: landy
 * @date: 2019/5/30 23:37
 * @description:
 */
@RunWith(SpringJUnit4ClassRunner.class) //Calling Spring Unit Test Class
@ContextConfiguration(classes = {
        SettingsConfiguration.class,  // common settings configuration
        DocClientConfiguration.class,
        ApplicationContextHelperConfiguration.class
}) //Load Spring configuration file
public class DocClientTest {

    @Test
    public void test() {
        DocClient.uploadFile(new File("C:\\06_temp\\psb.jpg"), "psb.jpg",false);
    }
}

Similar log information can also be displayed on the same server.

2019-07-21  00:04:26.704 [New I/O worker #4] INFO  org.fortune.doc.server.handler.UploadDocServerHandler - --srcFileName--psb.jpg
2019-07-21  00:04:26.705 [New I/O worker #4] INFO org.fortune.doc.server.handler.UploadDocServerHandler-file upload successfully, save the path as fortune\\\\\\\\\\\\\\\\\\\\721000426_6348.jpg
2019-07-21  00:04:26.705 [New I/O worker #4) DEBUG org.fortune.doc.server.handler.factory.DocServerHandler Factory-Execution result: {action":"uploadFile","code": 1,"filePath","fortune\\\\\\\ 1000426_6348.jpg","msg","file upload success":true}.
2019-07-21  00:04:26.705 [New I/O worker #4) INFO org.fortune.doc.server.support.DocServerHandler-File processing end....FileServerHandler Factory processing results: {"action", "uploadFile", "code": 1, "filePath": "fortune\\\\\\\\\\\\\
2019-07-21  00:04:26.706 [New I/O worker #4] INFO org.fortune.doc.server.support.DocServerHandler-arrives at the end of the file content and performs the corresponding file processing operations...
2019-07-21  00:05:50.539 [http-nio-8080-exec-6] DEBUG org.springframework.web.servlet.DispatcherServlet - GET "/doc-server/fortune//w//190721000426_6348.jpg", parameters={}
2019-07-21  00:05:50.540 [http-nio-8080-exec-6] DEBUG org.springframework.web.servlet.handler.SimpleUrlHandlerMapping - Mapped to org.springframework.web.servlet.resource.DefaultServletHttpRequestHandler@4fcd8ee1
2019-07-21  00:05:50.542 [http-nio-8080-exec-6] DEBUG org.springframework.web.servlet.DispatcherServlet - Completed 200 OK

Look at the server target directory, there is another file, which is shown as follows.

The path printed from the log information is the same. We can access the file by returning the address information.

5. Common Questions

5.1 Maven Dependence Problem

If some jar packages cannot be downloaded, you can download them manually and install them into the local warehouse manually by executing the maven command.

mvn install:install-file -DgroupId=com.fasterxml.jackson.core -DartifactId=jackson-core -Dversion=2.9.9.1 -Dpackaging=jar -Dfile=/path/to/file

Alternatively, if you host your own repository you can deploy the file there:)

mvn deploy:deploy-file -DgroupId=com.fasterxml.jackson.core -DartifactId=jackson-core -Dversion=2.9.9.1 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

Code engineering piece github, address: https://github.com/landy8530/...

Posted by nodehopper on Sat, 20 Jul 2019 21:21:59 -0700

Programmer Group