1, Demand
The file server uses HTTP protocol to provide external services. When accessing the file server through the browser, the user first checks the URL. If it fails, an error 403 is returned; If the current directory is opened by link through verification, each directory or file is displayed in the form of hyperlink, which can be accessed recursively and downloaded.
2, Key implementation code
① File server startup class
The channel processors to be added are as follows:
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast("http-decoder", new HttpRequestDecoder());
ch.pipeline().addLast("http-aggregator", new HttpObjectAggregator(65536));
ch.pipeline().addLast("http-encoder", new HttpResponseEncoder());
ch.pipeline().addLast("http-chunked", new ChunkedWriteHandler());
ch.pipeline().addLast("fileServerHandler", new HttpFileServerHandler(url));
}1) HttpRequestDecoder
Decodes {@link ByteBuf}s into {@link HttpRequest}s and {@link HttpContent}s.
It is responsible for decoding bytes into Http requests.
2) HttpObjectAggregator
A {@link ChannelHandler} that aggregates an {@link HttpMessage} and its following {@link HttpContent}s into a single {@link FullHttpRequest} or {@link FullHttpResponse} (depending on if it used to handle requests or responses)
It is responsible for assembling multiple httpmessages into a complete Http request or response. Whether it is assembled into a request or response depends on whether the content it processes is the content of the request or the content of the response. In fact, this can be judged by Inbound and Outbound. For the Server side, the request is received on the Inbound side and the response is returned on the Outbound side.
It is useful when you don't want to take care of HTTP messages whose transfer encoding is 'chunked'.
If the data returned by the Server to the Client, the specified transmission code is chunked. Then, the Server does not need to know the total length of the data sent to the Client. It is sent in blocks. Refer to the block transmission code
Be aware that you need to have the {@link HttpResponseEncoder} or {@link HttpRequestEncoder} before the {@link HttpObjectAggregator} in the {@link ChannelPipeline}.
Note that the HttpObjectAggregator channel processor must be placed after the httprequestedecoder or httprequesteencoder.
3) HttpResponseEncoder
After the Server processes the message, it needs to send a response to the Client. Then you need to encode the response into bytes and send it. Therefore, an HttpResponseEncoder processor is added.
4)ChunkedWriteHandler
A {@link ChannelHandler} that adds support for writing a large data stream asynchronously neither spending a lot of memory nor getting {@link OutOfMemoryError}.
The channel processor is mainly used to handle the case of large file transfer. When transferring large files, complex state management is required, and ChunkedWriteHandler implements this function.
5) HttpFileServerHandler
Custom channel processor, which aims to realize the business logic of file server.
After adding the channel processor, you need to start the server. The code is as follows:
ChannelFuture f = b.bind("localhost", port).sync();
f.channel().closeFuture().sync();Because all events in Netty are asynchronous, bind is an asynchronous operation, and channel closing is also an asynchronous operation. Therefore, ChannelFuture is used as a palcontroller to represent the results after the operation is executed.
Finally, close the event thread. The code is as follows:
bossGroup.shutdownGracefully(); workerGroup.shutdownGracefully();
② File processor class
HttpFileServerHandler.java is a custom channel processor used to implement the business logic of the HTTP file server. From the Handler added above, we can see that Netty has solved a lot of work for us in the implementation of HTTP file server, such as HttpRequestDecoder automatically helps us parse HTTP requests (parse bytes); Another example: the HttpObjectAggregator assembles the data in multiple HTTP requests into one. It is very useful when the response sent by the server does not know the length of the response in advance.
reference resources: HttpChunkAggregator analysis
The file processor is implemented by inheriting SimpleChannelInboundHandler. The code is as follows:
public class HttpFileServerHandler extends SimpleChannelInboundHandler<FullHttpRequest>{
private final String url;
public HttpFileServerHandler(String url) {
this.url = url;
}
@Override
protected void messageReceived(ChannelHandlerContext ctx,
FullHttpRequest request) throws Exception {
if(!request.decoderResult().isSuccess())
{
sendError(ctx, HttpResponseStatus.BAD_REQUEST);
return;
}
if(request.method() != HttpMethod.GET)
{
sendError(ctx, HttpResponseStatus.METHOD_NOT_ALLOWED);
return;
}When the server receives a message, the messageReceived method is automatically triggered. This method first judges the URL and only accepts GET requests.
After the relevant verification is passed, open the file through the RandomAccessFile class and construct the response.
RandomAccessFile randomAccessFile = null;
try{
randomAccessFile = new RandomAccessFile(file, "r");
}catch(FileNotFoundException fnfd){
sendError(ctx, HttpResponseStatus.NOT_FOUND);
return;
}
long fileLength = randomAccessFile.length();
HttpResponse response = new DefaultHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK);If the request contains "KEEP-ALIVE", the connection is not closed.
if(HttpHeaderUtil.isKeepAlive(request)){
response.headers().set(HttpHeaderNames.CONNECTION, HttpHeaderValues.KEEP_ALIVE);
}Send data
sendFileFuture = ctx.write(new ChunkedFile(randomAccessFile, 0, fileLength, 8192), ctx.newProgressivePromise());
sendFileFuture.addListener(new ChannelProgressiveFutureListener() {
@Override
public void operationComplete(ChannelProgressiveFuture future)
throws Exception {
System.out.println("Transfer complete.");
}ctx.write(response);After sending the data, because the transfer encoding: chunk mode is used to transmit the data, it is necessary to send a chunk with a length of 0 to mark the completion of data transmission.
reference resources: HTTP protocol header and keep alive mode
ChannelFuture lastContentFuture = ctx.writeAndFlush(LastHttpContent.EMPTY_LAST_CONTENT);
if(!HttpHeaderUtil.isKeepAlive(request))
lastContentFuture.addListener(ChannelFutureListener.CLOSE);
}Using keep alive can reduce the number of HTTP connection establishment. In HTTP 1.1, this option is enabled by default.
Connection: Keep-Alive When the server processes the request and generates a response, it also adds a header to the response: Connection: Keep-Alive When this is done, the socket connection is not closed as before, but kept open after sending the response. When the client sends another request, it reuses the same connection. The connection will continue to be reused until either the client or the server decides that the conversation is over, and one of them drops the connection.
When keep alive is used, when the Server processes the client's request and generates a response, add the connection: keep alive option in the response header to return the response to the client. At this time, the Socket connection will not be closed.
[if there is no keep alive, the Socket connection will be closed after an HTTP request response]
Since the connection has not been closed yet, when the client sends another request, the Socket connection will be reused until one of them drops the connection
For a discussion on keep alive, refer to:
The whole source code reference:
package httpFileServer;
import io.netty.bootstrap.ServerBootstrap;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelInitializer;
import io.netty.channel.EventLoopGroup;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.SocketChannel;
import io.netty.channel.socket.nio.NioServerSocketChannel;
import io.netty.handler.codec.http.HttpObjectAggregator;
import io.netty.handler.codec.http.HttpRequestDecoder;
import io.netty.handler.codec.http.HttpRequestEncoder;
import io.netty.handler.codec.http.HttpResponseEncoder;
import io.netty.handler.stream.ChunkedWriteHandler;
public class HttpFileServer {
private static final String DEFAULT_URL = "/src/";
public void run(final int port, final String url)throws Exception{
EventLoopGroup bossGroup = new NioEventLoopGroup();
EventLoopGroup workerGroup = new NioEventLoopGroup();
try{
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup).channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast("http-decoder", new HttpRequestDecoder());
ch.pipeline().addLast("http-aggregator", new HttpObjectAggregator(65536));
ch.pipeline().addLast("http-encoder", new HttpResponseEncoder());
ch.pipeline().addLast("http-chunked", new ChunkedWriteHandler());
ch.pipeline().addLast("fileServerHandler", new HttpFileServerHandler(url));
}
});
ChannelFuture f = b.bind("localhost", port).sync();
System.out.println("HTTP File server startup, the address is: " + "http://localhost:" + port + url);
f.channel().closeFuture().sync();
}finally{
bossGroup.shutdownGracefully();
workerGroup.shutdownGracefully();
}
}
public static void main(String[] args)throws Exception {
int port = 8888;
if(args.length > 0)
{
try{
port = Integer.parseInt(args[0]);
}catch(NumberFormatException e){
port = 8080;
}
}
String url = DEFAULT_URL;
if(args.length > 1)
url = args[1];
new HttpFileServer().run(port, url);
}
}
package httpFileServer;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.RandomAccessFile;
import java.io.UnsupportedEncodingException;
import java.net.URLDecoder;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.regex.Pattern;
import javax.activation.MimetypesFileTypeMap;
import javax.swing.text.html.MinimalHTMLWriter;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.Unpooled;
import io.netty.channel.Channel;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelFutureListener;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.ChannelProgressiveFuture;
import io.netty.channel.ChannelProgressiveFutureListener;
import io.netty.channel.SimpleChannelInboundHandler;
import io.netty.handler.codec.http.DefaultFullHttpResponse;
import io.netty.handler.codec.http.DefaultHttpResponse;
import io.netty.handler.codec.http.FullHttpRequest;
import io.netty.handler.codec.http.FullHttpResponse;
import io.netty.handler.codec.http.HttpHeaderNames;
import io.netty.handler.codec.http.HttpHeaderUtil;
import io.netty.handler.codec.http.HttpHeaderValues;
import io.netty.handler.codec.http.HttpHeaders;
import io.netty.handler.codec.http.HttpMessage;
import io.netty.handler.codec.http.HttpMethod;
import io.netty.handler.codec.http.HttpResponse;
import io.netty.handler.codec.http.HttpResponseStatus;
import io.netty.handler.codec.http.HttpVersion;
import io.netty.handler.codec.http.LastHttpContent;
import io.netty.handler.codec.http2.Http2Headers;
import io.netty.handler.stream.ChunkedFile;
import io.netty.util.CharsetUtil;
import io.netty.util.concurrent.Future;
import io.netty.util.concurrent.GenericFutureListener;
public class HttpFileServerHandler extends SimpleChannelInboundHandler<FullHttpRequest>{
private final String url;
public HttpFileServerHandler(String url) {
this.url = url;
}
@Override
protected void messageReceived(ChannelHandlerContext ctx,
FullHttpRequest request) throws Exception {
if(!request.decoderResult().isSuccess())
{
sendError(ctx, HttpResponseStatus.BAD_REQUEST);
return;
}
if(request.method() != HttpMethod.GET)
{
sendError(ctx, HttpResponseStatus.METHOD_NOT_ALLOWED);
return;
}
final String uri = request.uri();
final String path = sanitizeUri(uri);
if(path == null)
{
sendError(ctx, HttpResponseStatus.FORBIDDEN);
return;
}
File file = new File(path);
if(file.isHidden() || !file.exists())
{
sendError(ctx, HttpResponseStatus.NOT_FOUND);
return;
}
if(file.isDirectory())
{
if(uri.endsWith("/"))
{
sendListing(ctx, file);
}else{
sendRedirect(ctx, uri + "/");
}
return;
}
if(!file.isFile())
{
sendError(ctx, HttpResponseStatus.FORBIDDEN);
return;
}
RandomAccessFile randomAccessFile = null;
try{
randomAccessFile = new RandomAccessFile(file, "r");
}catch(FileNotFoundException fnfd){
sendError(ctx, HttpResponseStatus.NOT_FOUND);
return;
}
long fileLength = randomAccessFile.length();
HttpResponse response = new DefaultHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK);
HttpHeaderUtil.setContentLength(response, fileLength);
// setContentLength(response, fileLength);
setContentTypeHeader(response, file);
if(HttpHeaderUtil.isKeepAlive(request)){
response.headers().set(HttpHeaderNames.CONNECTION, HttpHeaderValues.KEEP_ALIVE);
}
ctx.write(response);
ChannelFuture sendFileFuture = null;
sendFileFuture = ctx.write(new ChunkedFile(randomAccessFile, 0, fileLength, 8192), ctx.newProgressivePromise());
sendFileFuture.addListener(new ChannelProgressiveFutureListener() {
@Override
public void operationComplete(ChannelProgressiveFuture future)
throws Exception {
System.out.println("Transfer complete.");
}
@Override
public void operationProgressed(ChannelProgressiveFuture future,
long progress, long total) throws Exception {
if(total < 0)
System.err.println("Transfer progress: " + progress);
else
System.err.println("Transfer progress: " + progress + "/" + total);
}
});
ChannelFuture lastContentFuture = ctx.writeAndFlush(LastHttpContent.EMPTY_LAST_CONTENT);
if(!HttpHeaderUtil.isKeepAlive(request))
lastContentFuture.addListener(ChannelFutureListener.CLOSE);
}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause)
throws Exception {
cause.printStackTrace();
if(ctx.channel().isActive())
sendError(ctx, HttpResponseStatus.INTERNAL_SERVER_ERROR);
}
private static final Pattern INSECURE_URI = Pattern.compile(".*[<>&\"].*");
private String sanitizeUri(String uri){
try{
uri = URLDecoder.decode(uri, "UTF-8");
}catch(UnsupportedEncodingException e){
try{
uri = URLDecoder.decode(uri, "ISO-8859-1");
}catch(UnsupportedEncodingException e1){
throw new Error();
}
}
if(!uri.startsWith(url))
return null;
if(!uri.startsWith("/"))
return null;
uri = uri.replace('/', File.separatorChar);
if(uri.contains(File.separator + '.') || uri.contains('.' + File.separator) || uri.startsWith(".") || uri.endsWith(".")
|| INSECURE_URI.matcher(uri).matches()){
return null;
}
return System.getProperty("user.dir") + File.separator + uri;
}
private static final Pattern ALLOWED_FILE_NAME = Pattern.compile("[A-Za-z0-9][-_A-Za-z0-9\\.]*");
private static void sendListing(ChannelHandlerContext ctx, File dir){
FullHttpResponse response = new DefaultFullHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK);
// response.headers().set("CONNECT_TYPE", "text/html;charset=UTF-8");
response.headers().set(HttpHeaderNames.CONTENT_TYPE, "text/html;charset=UTF-8");
String dirPath = dir.getPath();
StringBuilder buf = new StringBuilder();
buf.append("<!DOCTYPE html>\r\n");
buf.append("<html><head><title>");
buf.append(dirPath);
buf.append("catalogue:");
buf.append("</title></head><body>\r\n");
buf.append("<h3>");
buf.append(dirPath).append(" catalog:");
buf.append("</h3>\r\n");
buf.append("<ul>");
buf.append("<li>Link:<a href=\" ../\")..</a></li>\r\n");
for (File f : dir.listFiles()) {
if(f.isHidden() || !f.canRead()) {
continue;
}
String name = f.getName();
if (!ALLOWED_FILE_NAME.matcher(name).matches()) {
continue;
}
buf.append("<li>Link:<a href=\"");
buf.append(name);
buf.append("\">");
buf.append(name);
buf.append("</a></li>\r\n");
}
buf.append("</ul></body></html>\r\n");
ByteBuf buffer = Unpooled.copiedBuffer(buf,CharsetUtil.UTF_8);
response.content().writeBytes(buffer);
buffer.release();
ctx.writeAndFlush(response).addListener(ChannelFutureListener.CLOSE);
}
private static void sendRedirect(ChannelHandlerContext ctx, String newUri){
FullHttpResponse response = new DefaultFullHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.FOUND);
// response.headers().set("LOCATIN", newUri);
response.headers().set(HttpHeaderNames.LOCATION, newUri);
ctx.writeAndFlush(response).addListener(ChannelFutureListener.CLOSE);
}
private static void sendError(ChannelHandlerContext ctx, HttpResponseStatus status){
FullHttpResponse response = new DefaultFullHttpResponse(HttpVersion.HTTP_1_1, status,
Unpooled.copiedBuffer("Failure: " + status.toString() + "\r\n", CharsetUtil.UTF_8));
response.headers().set(HttpHeaderNames.CONTENT_TYPE, "text/html;charset=UTF-8");
ctx.writeAndFlush(response).addListener(ChannelFutureListener.CLOSE);
}
private static void setContentTypeHeader(HttpResponse response, File file){
MimetypesFileTypeMap mimetypesFileTypeMap = new MimetypesFileTypeMap();
response.headers().set(HttpHeaderNames.CONTENT_TYPE, mimetypesFileTypeMap.getContentType(file.getPath()));
}
}