node Implements Piecewise Download

Based on http Range Requests protocol, this paper realizes the function of fragmented download.

Use scenarios include browser-based streaming file fragment transfer, client-based fragmented download, etc.

principle

http through Range Requests related header, can negotiate with the server to achieve partial requests.

Here we will not elaborate on the specific content of the agreement. We can refer to these two articles and explain them in great detail.

The implementation process is pasted below.

Server-side code

The server is implemented with node:

const fs = require('fs');
const path = require('path');
const Koa = require('koa');

const app = new Koa();
const PATH = './resource';

app.use(async ctx => {
    const file = path.join(__dirname, `${PATH}${ctx.path}`);
    // 1. 404 examination
    try {
        fs.accessSync(file);
    } catch (e) {
        return ctx.response.status = 404;
    }
    const method = ctx.request.method;
    const { size } = fs.statSync(file);
    // 2. Respond to the head request and return the file size
    if ('HEAD' == method) {
        return ctx.set('Content-Length', size);
    }
    const range = ctx.headers['range'];
    // 3. Notify the browser that partial requests can be made
    if (!range) {
        return ctx.set('Accept-Ranges', 'bytes');
    }
    const { start, end } = getRange(range);
    // 4. Check the scope of requests
    if (start >= size || end >= size) {
        ctx.response.status = 416;
        return ctx.set('Content-Range', `bytes */${size}`);
    }
    // 5, 206 partial response
    ctx.response.status = 206;
    ctx.set('Accept-Ranges', 'bytes');
    ctx.set('Content-Range', `bytes ${start}-${end ? end : size - 1}/${size}`);
    ctx.body = fs.createReadStream(file, { start, end });
});

app.listen(3000, () => console.log('partial content server start'));

function getRange(range) {
    var match = /bytes=([0-9]*)-([0-9]*)/.exec(range);
    const requestRange = {};
    if (match) {
        if (match[1]) requestRange.start = Number(match[1]);
        if (match[2]) requestRange.end = Number(match[2]);
    }
    return requestRange;
}

The functional logic of the code implementation is roughly as follows:

Check the requested resource and respond 404 if it does not exist
For HEAD requests, return the resource size
If the GET request does not inform the range, return Content-Length to inform the browser that the fragmentation request can be made.
If the request sets the range, check whether the range is legal and return the legitimate rangge illegally
Everything is OK. Get the range section of the file and do the flow response.

The code is very simple. It's ok ay to implement the Range Requests protocol once. Of course, there is no complete implementation of the protocol here, but it has met the needs of the demonstration here.

Server code ok, with a browser demo to verify.

Browser examples

Range Requests are basically implemented in modern browsers, with audio tags as an example.

<html>
    <head>
        <title>Piecewise streaming transmission</title>
        <script type="text/javascript">
            function jump() {
                const player = document.getElementById('musicPlayer');
                // Play from 30s
                player.currentTime = 30;
            }
        </script>
    </head>
    <body>
        <audio id="musicPlayer" src="http:127.0.0.1:3000/source.mp3" controls></audio>
        <button onclick="jump()">Cut to 30 s</button>
    </body>
</html>

The ultimate effect is as follows:

Comparing the two graphs, when html is loaded and browsers request resources automatically, the header has Range: bytes=0-, which means loading resources from 0 byte; when clicking to play at 30 s, the header becomes Range: bytes=3145728-.

Similarly, with this server code, you can also implement a client to simulate the subpackage download.

node Subpackage Download

This example demonstrates that for a resource, the concurrent implementation of partial download, and then merge into a file.

This is also implemented with node:

import request from 'request';
import path from 'path';
import fs from 'fs';

const SINGLE = 1024 * 1000;
const SOURCE = 'http://127.0.0.1:3000/source.mp3';

request({
    method: 'HEAD',
    uri: SOURCE,
}, (err, res) => {
    if (err) return console.error(err);
    const file = path.join(__dirname, './download/source.mp3');
    try {
        fs.closeSync(fs.openSync(file, 'w'));
    } catch (err) {
        return console.error(err);
    }
    const size = Number(res.headers['content-length']);
    const length = parseInt(size / SINGLE);
    for (let i=0; i<length; i++) {
        let start = i * SINGLE;
        let end = i == length ? (i + 1) * SINGLE - 1 : size - 1;
        request({
            method: 'GET',
            uri: SOURCE,
            headers: {
                'range': `bytes=${start}-${end}`
            },
        }).on('response', (resp) => {
            const range = resp.headers['content-range'];
            const match = /bytes ([0-9]*)-([0-9]*)/.exec(range);
            start = match[1];
            end = match[2];
        }).pipe(fs.createWriteStream(file, {start, end}));
    }
});

The code is relatively simple, which is to open multiple http requests, concurrently download resources, and then write to the corresponding location of the file according to the content-range of the response.

Reference article:

Posted by goldages05 on Fri, 01 Feb 2019 15:51:15 -0800

Programmer Group