將 axios 請求通過管道傳輸到 pkgcloud 存儲上傳會導致“結束后寫入”錯誤

Question

我正在嘗試使用 axios 下載外部文件，然后使用 pkgcloud 將其通過管道傳輸到 openstack 對象存儲容器中。 上傳流似乎在文件完全下載之前就結束了。

async function upload(url, name, extension, container) {
    const uploadStream = storage.createClient({
        ...
    }).upload({
        container: container,
        remote: name + "." + extension,
    });

    await new Promise(async (resolve) => {
        const response = await axios({
            method: "GET",
            url: url,
            responseType: "stream"
        })

        response.data.pipe(uploadStream)

        response.data.on("end", () => {
            console.log("Download finished");
        })
        response.data.on("error", (error: Error) => {
            console.log("Download error")
            console.log(error);
        })
        uploadStream.on("finish", () => {
            console.log("Upload finished");
            resolve();
        })
        uploadStream.on("error", (error: Error) => {
            console.log("Upload error");
            console.log(error);
        })
    })
}

const url = "https://picsum.photos/id/566/600/600";

(async () => {
    await upload(url, "mountain", "jpg", "dummy_container")
    console.log("Promise resolved");
})()

根據控制台輸出的事件順序如下：

Download finished
Upload finished
Promise resolved
Upload error
Error: write after end
...

Answer 1

可以通過使用concat-stream包將下載流concat-stream到緩沖區中，然后將緩沖區流式傳輸到上傳中來解決此問題：

function upload(url, name, extension, container) {
    const uploadStream = storage.createClient({
        ...
    }).upload({
        container,
        remote: name + "." + extension,
    });

    axios({
        method: "GET",
        url,
        responseType: "stream",
    }).then((response) => {
        const concatStream = concat((buffer) => {
            const bufferStream = new Stream.PassThrough();
            bufferStream.end(buffer);
            bufferStream.pipe(uploadStream);
        })
        response.data.pipe(concatStream);
    })
}

const url = "https://picsum.photos/id/566/600/600";
upload(url, "mountain", "jpg", "dummy_container")

Answer 2

問題來自 pkgcloud 中的一個錯誤，它是如何為 OpenStack Storage 傳輸文件的。 在https://stackoverflow.com/a/60787524/4685491 中有解釋