使用google-cloud-storage将数据从gcs传输到s3

Question

I'm making a small app to export data from BigQuery to google-cloud-storage and then copy it into aws s3, but having trouble finding out how to do it in python.

I have already written the code in kotlin (because it was easiest for me, and reasons outside the scope of my question, we want it to run in python), and in kotlin the google sdk allows me to get an InputSteam from the Blob object, which i can then inject into the amazon s3 sdk's AmazonS3.putObject(String bucketName, String key, InputStream input, ObjectMetadata metadata) .

With the python sdk it seems i only have the options to download file to a file and as a string.

I would like (as i do in kotlin) to pass some object returned from the Blob object, into the AmazonS3.putObject() method, without having to save the content as a file first.

I am in no way a python pro, so i might have missed an obvious way of doing this.

Answer 1

I ended up with the following solution, as apparently download_to_filename downloads data into a file-like-object that the boto3 s3 client can handle.

This works just fine for smaller files, but as it buffers it all in memory, it could be problematic for larger files.

def copy_data_from_gcs_to_s3(gcs_bucket, gcs_filename, s3_bucket, s3_filename):
gcs_client = storage.Client(project="my-project")

bucket = gcs_client.get_bucket(gcs_bucket)
blob = bucket.blob(gcs_filename)

data = BytesIO()
blob.download_to_file(data)
data.seek(0)

s3 = boto3.client("s3")
s3.upload_fileobj(data, s3_bucket, s3_filename)

If anyone has information/knowledge about something other than BytesIO to handle the data (fx. so i can stream the data directly into s3, without having to buffer it in memory on the host-machine) it would be very much appreciated.

Answer 2

Google-resumable-media can be used to download file through chunks from GCS and smart_open to upload them to S3. This way you don't need to download whole file into memory. Also there is an similar question that addresses this issue Can you upload to S3 using a stream rather than a local file?