How do I send a compressed string to the client from Node running Koa?

Question

I want to send a very large gzipped JSON string to the client.

I have tried many methods illustrated in the koa and node docs, but I can't wrap my head around the koa expectations for streaming/piping to its context.response.body object.

I've tried

const buf = Buffer.from(json, 'utf-8');   
zlib.gzip(buf, function (_, result) {  
    ctx.request.body = result;                     
});

based on this stackoverflow question.

Also:

var input = new Buffer(json, 'utf8')
ctx.request.body = zlib.deflate(input)

as well as

var input = new Buffer(json, 'utf8')
ctx.request.body = zlib.deflate(input).toString('utf8')

based on this stackoverflow question.

I've also tried using archiver:

const archive = archiver.create('zip', {});
archive.pipe(ctx.body);
archive.append(JSON.stringify(json), { name: 'libraries.json'});
archive.finalize();

Most cases, either the client receives a malformed zip/file, or it receives index.html as a file.

I've also tried using koa-compress , using this stackoverflow question as a guide.

app.use(compress({
    filter: function (content_type: any) {
        return /text/i.test(content_type);
    },
    threshold: 2048,
    flush: require('zlib').Z_SYNC_FLUSH
}));

...

ctx.response.type = 'application/json';
ctx.body = json;
ctx.compress = true;
ctx.status = 200;

In this case, the client does download the json as a .json successfully, but nothing in the inspector indicates to me that it was actually gzipped:

HTTP/1.1 200 OK
X-Powered-By: Express
vary: Accept-Encoding
last-modified: Thu, 01 Nov 2018 21:26:55 GMT
cache-control: max-age=0
content-type: application/json; charset=utf-8
content-length: 4767074
date: Tue, 11 Dec 2018 23:44:11 GMT

The client is trying the "create blob" trick to download the file:

const xhr = new XMLHttpRequest();
const default_headers = {
    'Content-Type': 'application/json',
};
xhr.open('GET', `${API_URL}${url}`);
for (let header in default_headers) {
    xhr.setRequestHeader(header, default_headers[header]);
}
xhr.onload = function() {
    if (this.status === 200) {
        // Create a new Blob object using the response data of the onload object
        var blob = new Blob([this.response], { type: 'application/json' });
        // Create a link element, hide it, direct it towards the blob, and then 'click' it programatically
        let a = document.createElement('a');
        a.style = 'display: none';
        document.body.appendChild(a);
        // Create a DOMString representing the blob and point the link element towards it
        let url = window.URL.createObjectURL(blob);
        a.href = url;
        a.download = 'libraries.json';
        // have also tried libraries.zip, .gzip, etc
        // programatically click the link to trigger the download
        a.click();
        // release the reference to the file by revoking the Object URL
        window.URL.revokeObjectURL(url);
    }
};
xhr.responseType = 'blob';
xhr.send();

I have also tried just having the client make the request "vanilla," without any blob tricks. Then, there is no "file download" dialog launched.

I expect there is a fundamental misunderstanding on my part here about AJAX and file downloads.

How do I allow a browser client to request, and then download, a (very large) compressed JSON from a node server that runs Koa ?

Answer 1

The lazy way is to let someone else do the work for you:

https://www.npmjs.com/package/koa-compress https://github.com/koajs/compress

As easy as this:

const responseBodyCompressor = require("koa-compress");
...
app.use( responseBodyCompressor() )
...