如何在不进行任何手动操作的情况下将数据从 Google 云存储传输到 S3？

Question

I want to transfer data from my cloud storage bucket to S3. I did a lot of research. I found an interesting article in which this can be done using gsutil command. Again it requires a manual activity. Is there any way to do this without any manual activity or like any possibility to run this gsutil command through Java API?

I have created a VM instance on my Google cloud project.I have configured the boto file which has my AWS credentials.Using gsutil command I was able to copy my data from cloud storage to s3.I want to automate this step

Answer 1

gsutil can work with both Google Storage and S3.

gsutil rsync -d -r gs://my-gs-bucket s3://my-s3-bucket

You just need to configure it with both - Google and your AWS S3 credentials. gsutil would use credentials from ~/.boto file.

There are multiple ways to automate shell command execution. From Java's Runtime.exec() to Google Cloud Scheduler (your cloud's cron)

Answer 2

Fully automated and easy to use way would be Google Storage Transfer . It would let you select the source and target buckets in either S3 or GCS and set shedule such as "run daily at 1am"

Answer 3

Skyplane is a much faster alternative for transferring data between clouds (up to 110x for large files). You can transfer data with the command:

skyplane cp -r s3://aws-bucket-name/ gcs://google-bucket-name/