Google cloud platform to S3/Redshift

Question

Is there anyway programmatically in Python to ingest data from Google cloud platform in S3? My ultimate aim is to get data from Google Big Query to Redshift.

Answer 1

You can probably use gsutil to copy data from a Google Cloud Storage bucket to an Amazon s3 bucket, using a command like this

gsutil -m rsync -rd gs://gcs-bucket s3://s3-bucket

no need for python code.

Answer 2

您还可以查看https://rclone.org/，它支持跨许多云存储（包括Google Storage和AWS S3）复制数据。