How to upload HDF5 file directly to S3 bucket in Python

Question

I want to upload a HDF5 file created with h5py to S3 bucket without saving locally using boto3.

This solution uses pickle.dumps and pickle.loads and other solutions I have found, store the file locally which I like to avoid.

Answer 1

You can use io.BytesIO() to and put_object as illustrated here 6 . Hope this helps. Even in this case, you'd have to 'store' the data locally(though 'in memory'). You could also create a tempfile.TemporaryFile and then upload your file with put_object . I don't think you can stream to an S3 Buckets in the sense that the local data would be discarded as it is uploaded to the Bucket.