简体   繁体   English

发送ArrayBuffer到S3放入signedURL

[英]Send ArrayBuffer to S3 put to signedURL

I am progressively loading a file into a buffer, the buffer is valid, but the browser crashes when the ArrayBuffer is finished loading the file into it. 我正在逐步将文件加载到缓冲区中,缓冲区有效,但是当ArrayBuffer完成将文件加载到其中时浏览器崩溃。 What I need to do is to be able to send the pieces of the buffer buf = this.concatBuffers(buf, buffer); 我需要做的是能够发送缓冲区块buf = this.concatBuffers(buf, buffer); to the axios PUT request so I can progressively upload the file to s3, rather than load it into a single variable returned by the promise (as the memory gets exceeded). 到axios PUT请求所以我可以逐步将文件上传到s3,而不是将它加载到promise返回的单个变量中(因为超出了内存)。

How do I modify the link between readFileAsBuffer and the uploadFileToS3 method to do this? 如何修改readFileAsBuffer和uploadFileToS3方法之间的链接来执行此操作?

This is my code so you can follow the process. 这是我的代码,因此您可以按照该过程进行操作。

concatTypedArrays = (a, b) => {
  const c = new a.constructor(a.length + b.length);
  c.set(a, 0);
  c.set(b, a.length);
  return c;
};

concatBuffers = (a, b) =>
  this.concatTypedArrays(
    new Uint8Array(a.buffer || a),
    new Uint8Array(b.buffer || b),
  ).buffer;

readFileAsBuffer = file =>
  new Promise((resolve, reject) => {
    const fileReader = new FileReader();
    fileReader.file = file;
    let buf = new ArrayBuffer();
    const fileChunks = new FileChunker(file, 2097152);

    fileReader.readAsArrayBuffer(fileChunks.blob());

    fileReader.onload = e => {
      this.onProgress(fileChunks);
      const buffer = e.target.result;
      buf = this.concatBuffers(buf, buffer);

      if (fileChunks.hasNext()) {
        fileChunks.next();
        fileReader.readAsArrayBuffer(fileChunks.blob());
        return;
      }

      resolve(buf);
    };

    fileReader.onerror = err => {
      reject(err);
    };
  });

uploadFileToS3 = fileObject => {
  new Promise((resolve, reject) => {
    const decodedURL = decodeURIComponent(fileObject.signedURL);

    this.readFileAsBuffer(fileObject.fileRef).then(fileBuffer => {
      console.log(fileBuffer);
      axios
        .put(decodedURL, fileBuffer, {
          headers: {
            'Content-Type': fileObject.mime,
            'Content-MD5': fileObject.checksum,
            'Content-Encoding': 'UTF-8',
            'x-amz-acl': 'private',
          },
          onUploadProgress: progressEvent => {
            const { loaded, total } = progressEvent;
            const uploadPercentage = parseInt(
              Math.round((loaded * 100) / total),
              10,
            );
            this.setState({ uploadProgress: uploadPercentage });
            console.log(`${uploadPercentage}%`);

            if (uploadPercentage === 100) {
              console.log('complete');
            }
          },
        })
        .then(response => {
          resolve(response.data);
        })
        .catch(error => {
          reject(error);
        });
    });
  });
};

uploadAllFilesToS3 = () => {
  const { files } = this.state;
  new Promise((resolve, reject) => {
    Object.keys(files).map(idx => {
      this.uploadFileToS3(files[idx])
        .then(response => {
          this.setState({ files: [] });
          resolve(response.data);
        })
        .catch(error => {
          reject(error);
        });
    });
  });
};

calcFileMD5 = file =>
  new Promise((resolve, reject) => {
    const fileReader = new FileReader();
    fileReader.file = file;
    const spark = new SparkMD5.ArrayBuffer();
    const fileChunks = new FileChunker(file, 2097152);

    fileReader.readAsArrayBuffer(fileChunks.blob());

    fileReader.onload = e => {
      this.onProgress(fileChunks);
      const buffer = e.target.result;
      spark.append(buffer);

      if (fileChunks.hasNext()) {
        fileChunks.next();
        fileReader.readAsArrayBuffer(fileChunks.blob());
        return;
      }

      const hash = spark.end();
      const checksumAWS = Buffer.from(hash, 'hex').toString('base64');

      resolve(checksumAWS);
    };

    fileReader.onerror = err => {
      reject(err);
    };
  });

I ended up not needing to create my own Buffer of the file, instead if I post the fileReference returned by the input directly to axios (or xhr) the request automatically chunked the upload. 我最终不需要创建自己的文件缓冲区,而是如果我将输入返回的fileReference直接发布到axios(或xhr),请求会自动将上传文件分块。

Initially I could only make it work with XMLHttpRequest, but I quickly found a way to wrap this around axios which neatens the logic. 最初我只能使它与XMLHttpRequest一起工作,但我很快就找到了一种方法来围绕axios来包装逻辑。

XMLHttpRequest
const xhr = createCORSRequest('PUT', url);

if (!xhr) {
  console.log('CORS not supported');
} else {
  xhr.onload = function(){
    if(xhr.status == 200) {
      console.log('completed');
    } else {
      console.log('Upload error: ' + xhr.status);
    }
  };

  xhr.onerror = function(err) {
    console.log(err)
  };

  xhr.upload.onprogress = function(progressEvent){
   console.log(progressEvent);
  };

  xhr.setRequestHeader('Content-Type', file.type);
  xhr.setRequestHeader('Content-MD5', md5_base64_binary);
  xhr.setRequestHeader('Content-Encoding', 'UTF-8');
  xhr.setRequestHeader('x-amz-acl', 'private');
  xhr.send(file);
}

Or using axios; 或者使用axios;

uploadFileToS3 = fileObject => {
  return new Promise((resolve, reject) => {
    const { enqueueSnackbar } = this.props;
    const decodedURL = decodeURIComponent(fileObject.signedURL);

    axios
      .put(decodedURL, fileObject.fileRef, {
        headers: {
          'Content-Type': fileObject.mime,
          'Content-MD5': fileObject.checksum,
          'Content-Encoding': 'UTF-8',
          'x-amz-acl': 'private',
        },
        onUploadProgress: progressEvent => {
          const { loaded, total } = progressEvent;
          const uploadPercentage = parseInt(
            Math.round((loaded * 100) / total),
            10,
          );
          this.setState({ uploadProgress: uploadPercentage });
        },
      })
      .then(response => {
        resolve(response.data);
      })
      .catch(error => {
        reject(error);
      });
  });
};

Have you tried uploading your file using formData? 您是否尝试使用formData上传文件? Let the browser deal with file reading. 让浏览器处理文件读取。

const data = new FormData()
data.append('file', file)

axios.put(decodedURL, data, ....)

Another option is to use axios https://github.com/axios/axios#request-config transformRequest property. 另一种选择是使用axios https://github.com/axios/axios#request-config transformRequest属性。 And call for file reading there. 并呼吁在那里阅读文件。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM