如何生成多個 CSV Azure Blob Storage V2 的 TAR Zip(.tar.gz) 並使用 SharpZipLib 庫在 Blob 中輸出結果文件

Question

我需要在名為input 的容器內的 Azure blob storageV2 中使用 TarZip(.tar.gz) 多個 CSV 文件，並使用 Azure 函數在 C# 中使用 SharpZipLib 庫將結果文件保存在另一個容器輸出中。 單個文件的 CSV 文件大小最多可達 3 GB。

它的工作原理是從工作目錄中的 blob 下載文件，然后 tar zip 並上傳 blob 中的 tar zip 文件。 我想在不直接在 Blob 上進行壓縮的情況下下載它。 由於文件大小要高得多，約為 4 GB。

        public static async void tar()
        {
            // define blobs you need to use
            string connectionString = "XXXX";
            string Azure_container_name = "input";
            List<string> blobs = ListAllBlobsName(connectionString, Azure_container_name);
            
            
            BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
            var sourceContainer = blobServiceClient.GetBlobContainerClient("input");
            var desContainer = blobServiceClient.GetBlobContainerClient("output");
            var desBlob = desContainer.GetBlockBlobClient("file.tar.gz");
            var options = new BlockBlobOpenWriteOptions
            {
                HttpHeaders = new BlobHttpHeaders
                {
                    ContentType = MimeMapping.GetMimeMapping("file.tar.gz"),
                },
            };
            using (var outStream = desBlob.OpenWriteAsync(true, options).GetAwaiter().GetResult())
            using (TarOutputStream tarOutputStream = new TarOutputStream(outStream, Encoding.UTF8))
            {

                foreach (var blob in blobs)
                {
                    var source = sourceContainer.GetBlobClient(blob);
                    Console.WriteLine("Adding file "+blob + " in tar zip");
                    Azure.Storage.Blobs.Models.BlobProperties properties = source.GetPropertiesAsync().GetAwaiter().GetResult();
                    var entry = TarEntry.CreateTarEntry(blob);
                    entry.Size = properties.ContentLength;
                    tarOutputStream.PutNextEntry(entry);
                     source.DownloadToAsync(tarOutputStream).GetAwaiter().GetResult();
                    tarOutputStream.CloseEntry();
                    Console.WriteLine("Added file " + blob + " in tar zip");
                    Console.WriteLine();
                }
                tarOutputStream.Finish();
                tarOutputStream.Close();
            }
        }

        

Answer 1

關於該問題，請參考以下代碼

 // define blobs you need to use
            string[] blobs = { "",... };
            string connectionString = "";
            BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
            var sourceContainer = blobServiceClient.GetBlobContainerClient("input");
            var desContainer = blobServiceClient.GetBlobContainerClient("output");
            var desBlob= desContainer.GetBlockBlobClient( "csv.tar.gz");
            var options = new BlockBlobOpenWriteOptions {
                HttpHeaders = new BlobHttpHeaders {
                    ContentType = MimeMapping.GetMimeMapping("csv.tar.gz"),
                },
            };

            using (var outStream = await desBlob.OpenWriteAsync(true, options).ConfigureAwait(false))
            using (TarOutputStream tarOutputStream = new TarOutputStream(outStream, Encoding.UTF8)) {
                
                foreach (var blob in blobs) {
                    var source =sourceContainer.GetBlobClient(blob);

                    BlobProperties properties = await source.GetPropertiesAsync().ConfigureAwait(false);
                    var entry = TarEntry.CreateTarEntry(blob);
                    entry.Size = properties.ContentLength;
                    tarOutputStream.PutNextEntry(entry);
                    await source.DownloadToAsync(tarOutputStream);
                    tarOutputStream.CloseEntry();
                }
                tarOutputStream.Finish();
                tarOutputStream.Close();
            }