通过http下载并使用Node.js流传输到Google Cloud存储

Question

I am fighting to get a download to directly stream to GCS without saving to the file system, see below snippet.

const { Storage } = require('@google-cloud/storage');
const http = require('http');
const storage = new Storage();
const fs = require("fs");
const bucketName = 'BUCKETNAMEHERE';
const blobName = 'image.jpg';
const bucket = storage.bucket(bucketName);
const blob = bucket.file(blobName);

const streamDownload = () => {
    http.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg")
        .pipe(blob.createWriteStream({
            metadata: {
                contentType: 'image/jpg'
            }
        }))
        .on("error", (err) => {
            console.error(`error occurred`);
        })
        .on('finish', () => {
            console.info(`success`);
        });
};

On finish is never triggered. No output whatsoever. I can stream http.get to a local file without problems, so that part seems to be OK.

What also works is streaming from the local filesystem to GCS, like below :

const streamFs = () => {
    fs.createReadStream('/path/to/mqdefault.jpg')
        .pipe(blob.createWriteStream({
            metadata: {
                contentType: 'image/jpg'
            }
        }))
        .on("error", (err) => {
            console.error(`error occurred`);
        })
        .on('finish', () => {
            console.info(`success`);
        });
};

The second snippet logs 'success' and the file is present on the bucket.

Both http.get and fs.createReadStream create a read stream.

What am I doing wrong here ?

Answer 1

Switching over to the request library does work :

const request = require('request');
const streamDownload = () => {
    request.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg")
        .pipe(blob.createWriteStream({
            metadata: {
                contentType: 'image/jpg'
            }
        }))
        .on("error", (err) => {
            console.error(`error occurred`);
        })
        .on('finish', () => {
            console.info(`success`);
        });
};

Still unsure why http library does not.

Answer 2

the request library works with promises whereas the http lib needs a callback :

const streamDownload = () => {

  var stream = blob.createWriteStream({
    metadata: {
        contentType: 'image/jpg'
    }
  })

  http.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg", function(res){
        stream.on("error", (err) => {
            console.error(`error occurred`);
        })
        stream.on('finish', () => {
            console.info(`success`);
        });
        res.pipe(stream)            
    })
}