Utility for copying bucket data from one project to another in NodeJs

Question

I need to copy data from one bucket in Project A in gCloud to another bucket in Project B. Any utility if present in NodeJs to do this?

Answer 1

You might be tempted to get a list of blobs inside the bucket, download them, and upload them to another storage bucket (this is what gsuilt cp -m gs://origin_bucket/** gs://destination_bucket/ does).

The problem with this approach is you will consume CPU (on your side), and will take some time.

If you want to move all data from one bucket to another one, the best way to do this is using the Storage Transfer Service .

With the Storage Transfer Service you just tell the origin and destination buckets, and optionally a schedule... and google will do the operation much faster than you can do it yourself.

Also remember that the source and destination buckets can be GCS buckets, S3, or Azure Storage buckets, too.

Take a look at google-provided Node Sample Code for Storage Transfer Service .

If you just want to transfer some files, the Storage Transfer Service has a (in beta as Feb 2022) feature that allows you to specify a manifest file (a CSV file stored in a gcs bucket). See: Transfer specific files or objects using a Manifest

Answer 2

You can use the Cloud Storage Client Libraries . To copy an object in one of your Cloud Storage buckets, see this sample code:

const srcBucketName = 'your-source-bucket';
const srcFilename = 'your-file-name';
const destBucketName = 'target-file-bucket';
const destFileName = 'target-file-name';

// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');

// Creates a client
const storage = new Storage();

async function copyFile() {
  // Copies the file to the other bucket
  await storage
    .bucket(srcBucketName)
    .file(srcFilename)
    .copy(storage.bucket(destBucketName).file(destFileName));

  console.log(
    `gs://${srcBucketName}/${srcFilename} copied to gs://${destBucketName}/${destFileName}`
  );
}

copyFile().catch(console.error);

Additionally, you need to ensure that you have been assigned a role with the necessary permissions from the other project.