[英]Browser Javascript: Compress Json to gzip and upload to S3 presigned URL
Any advice would be appreciated. 任何意见,将不胜感激。 I've got a json variable in my web application that I'd like to gzip and upload to S3 through a presigned URL.
我的Web应用程序中有一个json变量,我想对其进行gzip压缩并通过预先签名的URL上传到S3。
I'm able to upload JSON successfully, but I fail to gzip the JSON and then upload it. 我能够成功上传JSON,但无法对JSON进行gzip压缩后再上传。
The three separate different ways I've tried to build the gzipped json is: 我尝试构建gzip json的三种不同方法是:
// example json
const someJson = { testOne: 'a', testTwo: 'b' };
// Attempt one
const stringUtf16 = JSON.stringify(someJson);
const resultAsBinString = pako.gzip(stringUtf16);
// Attempt two
const stringUtf16 = JSON.stringify(someJson);
const resultAsBinString = pako.gzip(stringUtf16, { to: 'string' });
// Attempt three
const stringUtf16ThatWeNeedInUtf8 = JSON.stringify(someJson);
const stringUtf8 = unescape(encodeURIComponent(stringUtf16ThatWeNeedInUtf8));
const resultAsBinString = pako.gzip(stringUtf8);
For each attempt, I uploaded the resultAsBinString through Angular's HTTP client, with the headers Content-Type: 'application/x-gzip' and Content-Encoding: 'gzip' 对于每次尝试,我都通过Angular的HTTP客户端上传了resultAsBinString,标题为Content-Type:“ application / x-gzip”和Content-Encoding:“ gzip”
But when (and if, oftentimes it gives a network error) the file is afterwards downloaded from S3, when trying to unzip with gzip or gunzip in the terminal, an error message is given: 'not in gzip format' 但是,当随后(如果通常会出现网络错误)从S3下载文件时,尝试在终端中使用gzip或gunzip解压缩时,会出现错误消息:“非gzip格式”
Sources I've tried to follow: 我尝试遵循的资料来源:
https://github.com/nodeca/pako/issues/55 https://github.com/nodeca/pako/blob/master/examples/browser.html https://github.com/nodeca/pako/issues/55 https://github.com/nodeca/pako/blob/master/examples/browser.html
Setting Content-Encoding: gzip
is not correct, if you expect the payload to remain gzipped after the download. 设置
Content-Encoding: gzip
不正确,如果您希望下载后的负载仍然保持压缩状态。 This is only used when you want the browser to transparently decode the gzip encoding -- such as when serving gzipped HTML, JavaScript, CSS, etc. 仅当您希望浏览器透明地解码gzip编码时(例如,提供gzip压缩的HTML,JavaScript,CSS等时), 才使用此功能。
If Content-Encoding: gzip
is used, then Content-Type
should be set to match the actual payload, such as Content-Type: application/json
. 如果使用
Content-Encoding: gzip
,则应将Content-Type
设置为匹配实际的有效负载,例如Content-Type: application/json
。
If Content-Type: application/x-gzip
is used, the Content-Encoding
should not be used, unless you're using a different sort of compression to re-compress the gzip payload (unlikely). 如果使用
Content-Type: application/x-gzip
,则不应使用Content-Encoding
,除非您使用另一种压缩方式重新压缩gzip有效负载(不太可能)。
Content-Type: application/x-gzip
combined with Content-Encoding: gzip
means you have wrapped a gzipped file inside another layer of gzip compression and you want the outer layer removed by the browser, which is not something you would ever do in practice. Content-Type: application/x-gzip
与Content-Encoding: gzip
组合Content-Encoding: gzip
表示您已将gzip压缩文件包装在gzip压缩的另一层中,并且希望浏览器删除外层,这实际上是您从未做过的。
The following process worked for me: 以下过程对我有用:
Generate the presigned URL with Content-Type: 'application/json'. 生成内容类型为'application / json'的预签名URL。 The provided filename should include the .gz at the end.
提供的文件名应在末尾包含.gz。 In the returned presigned URL, scanning through the URL should verify the Content Type is application/json.
在返回的预签名URL中,浏览URL应验证内容类型为application / json。
Because I'm certain my JSON contains no strings that would break the conversion to UTF-8, I then do the following (code in Angular, but it conveys the structure): 因为我确定我的JSON不包含会中断到UTF-8转换的字符串,所以我执行以下操作(Angular中的代码,但它传达了结构):
const headers = new HttpHeaders({
'Content-Type': 'application/json',
'Content-Encoding': 'gzip'
}); //1
const httpOptions = {
headers: headers
};
const str = JSON.stringify(geoJson); //2
const utf8Data = unescape(encodeURIComponent(str)); //3
const geoJsonGz = pako.gzip(utf8Data); //4
const gzippedBlob = new Blob([geoJsonGz]); //5
upload = this.httpClient.put(presignedUploadUrl, gzippedBlob, httpOptions); //6
Steps followed in the code: 代码中遵循的步骤:
You can then download the gzipped file from S3 (it should automatically be unzipped by the browser) and open it to verify that it contains the same results. 然后,您可以从S3下载gzip压缩文件(浏览器应自动将其解压缩)并打开该文件以验证其是否包含相同的结果。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.