如何将.csv 文件合并到谷歌云存储桶的几个分片中？

Question

I have moved a public dataset available as a Public Google Storage Bucket into my own bucket. The file size is about 10 GB. When the data moved, the file was split into about 47 shards, all compressed. I am unable to combine them into one file. How can I combine them?

Information given on the following link does not help much:

https://cloud.google.com/storage/docs/gsutil/commands/compose

My bucket looks like this:

Any help will be appreciated.

Answer 1

I propose you to use Cloud Build . It's not the most obvious solution, but it's serverless and cheap. Perfect for your 1 time use case. here what I propose to perform

steps:
- name: 'gcr.io/cloud-builders/gsutil'
  entrypoint: "bash"
  args: 
    - -c
    - |
       # copy all your files locally
       gsutil -m cp gs://311_nyc/311* .

       # Uncompress your file
       # I don't know your compression method? gunzip?

       # append your file in a merged file. Delete the files after the merge.
       for file in $(ls -1 311* ); do cat $file >> merged; rm $file; done

       # Copy the file to the destination bucket
       gsutil cp merged gs://myDestinationBucket/myName.csv

options:
# Use 1Tb of disk for getting all the files in the same time on the same server. 
# I didn't understand is the 10Gb is per uncompressed file or the total size. 
# If it's the total file size, I think that this option is useless
 diskSizeGb: 1000

# Optionally extend the default 10 minutes timeout if it takes too much time.
 timeout: 660s

Answer 2

Combine using nodejs

const { Storage } = require('@google-cloud/storage');
await storage.bucket(bucketName).combine(sourceFilenameList, destFilename)

Answer 3

If anyone else needs something like this.

This should do what you need:

Command line utility: https://github.com/tcwicks/DataUtilities

download latest release, unzip and use

Explanation / guide: https://medium.com/@TCWicks/merge-multiple-csv-flat-files-exported-from-bigquery-redshift-etc-d10aa0a36826

Hope someone finds it useful.