[Three Asynchronous HTTP Programming] 2. Streaming HTTP Response

Keywords: Programming Java encoding Big Data Windows

Standard Response and Content-Length Header

Since HTTP 1.1, in order to process multiple HTTP requests and responses in a link, the server must return the appropriate Content-Length value along with the response.

By default, you do not need to return the Content-Length header for simple requests, such as:

def index = Action {
    Ok("Hello World!")
}

Because the response to be sent is simple and clear, Play calculates the content size and generates the appropriate header.

Note: Text type content is not as simple as it seems, because the Content-Length header must convert text to binary in the correct encoding format.

In fact, we've seen the use of play.api.http.HttpEntity to define responses before:

def action = Action {
    Result {
        header = ResponseHeader(200, Map.empty)
        body = HttpEntity.Strict(ByteSting("Hello world"), Some("text/plain"))
    }
}

This means that Play must load the entire response content into memory and calculate its length.

Send Big Data

When the response data is too large, the above method is not feasible. Let's take sending a large file to the client as an example.

We use Source[ByteString,] to construct response content:

val file = new java.io.File("/tmp/fileToServe.pdf")
val path: java.nio.file.Path = file.toPath
val source: Source[ByteString, _] = FileIO.fromPath(path)

It looks quite simple. Next, we construct the response body with the streaming HttpEntity:

def streamed = Action {
    val file = new java.io.File("/tmp/fileToServer.pdf")
    val path: java.nio.file.Path = file.toPath
    val source: Source[ByteString, _] = FileIO.fromPath(path)

    Result {
        header = ResponseHeader(200, Map.empty)
        body = HttpEntity.Streamed(source, None, Some("application/pdf"))
    }
}

There is a problem here. We did not set Content-Length. Play will try to calculate, so it loads the entire file into memory and calculates the response size.

We don't want this when the volume of the document is too large. So you need to set Content-Length manually.

def streamedWithContentLength = Action {

  val file = new java.io.File("/tmp/fileToServe.pdf")
  val path: java.nio.file.Path = file.toPath
  val source: Source[ByteString, _] = FileIO.fromPath(path)

  val contentLength = Some(file.length())

  Result(
    header = ResponseHeader(200, Map.empty),
    body = HttpEntity.Streamed(source, contentLength, Some("application/pdf"))
  )
}

This method realizes the lazy loading of body data, divides the response into several blocks and copies them into HTTP responses.

File service

Play provides simple helpers to deal with common tasks for local files.

def file = Action {
    Ok.SendFile(new java.io.File("/tmp/fileToServe.pdf"))
}

This method generates a Content-Type response header based on the file name and adds a Content-Disposition header to indicate how the browser handles it. By default, an inline segment: Content-Disposition: inline; filename=fileToServe.pdf will be added to Disposition.

You can also customize the file name:

def fileWithName = Action {
    Ok.sendFile(
        content = new java.io.File("/tmp/fileToServe.pdf"),
        fileName = _ => "termsOfService.pdf"
    )
}

If you want to use the form of an attachment:

def fileAttachment = Action {
  Ok.sendFile(
    content = new java.io.File("/tmp/fileToServe.pdf"),
    inline = false
  )
}

Now you don't need to specify a file name, because browsers don't download files but display them directly in windows. This is useful in browser native support media formats, such as text, HTML or images.

response Blocking

Now we can calculate the length of streaming content before streaming it. Everything works well. But what if the content itself is dynamically generated without an existing length value?

For this kind of response, we need to use Chunked transfer encoding.

Block coding is proposed by HTTP 1.1 for web server to transmit a series of block data. It uses the HTTP header of Transfer-Encoding instead of the Content-Length header. Because the Content-Length header is no longer available, the server does not need to know in advance the length of the content to be transferred to the client. Now the server can generate content dynamically.

The size of each block is transmitted before the block, so the client can distinguish whether the current block has ended. The completion of all data transmission will be marked by a block with a length of 0.

More information about Chunked transger coding is available Wikipedia.

We can now transfer data in real time, that is, we can transfer data as soon as it is available. The resulting drawback is that the browser does not know the size of the content, so it cannot display a download progress bar.

Now suppose we have an InputStream that generates data dynamically. First we need to get stream:

val data = getDataStream
val dataContent: Source[ByteString, _] = StreamConverters.fromInputStream(() => data)

Now we can use Ok.chunked:

def chunkedFromSource = Action {
    val source = Source.apply(List("kiki", "foo", "bar"))
    Ok.chunked(source)
}

Let's look at the response returned by the server:

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked

4
kiki
3
foo
3
bar
0

We generated three data blocks and ended the response with a blank block.

Posted by johlwiler on Sun, 19 May 2019 16:00:43 -0700