![](/img/trans.png)
[英]Convert Pandas Dataframe to Image and Upload Directly to S3 Bucket Without Saving Locally Using Python
[英]upload a dataframe as a zipped csv directly to s3 without saving it on the local machine
如何将数据帧作为压缩的csv 上传到 S3 存储桶中,而无需先将其保存在本地计算机上?
我已经使用以下方法连接到该存储桶:
self.s3_output = S3(bucket_name='test-bucket', bucket_subfolder='')
我们可以使用标准库中的 BytesIO 和 zipfile 制作一个类似文件的对象。
# 3.7
from io import BytesIO
import zipfile
# .to_csv returns a string when called with no args
s = df.to_csv()
with zipfile.ZipFile(BytesIO(), mode="w",) as z:
z.writestr("df.csv", s)
# upload file here
您需要参考upload_fileobj以自定义上传的行为方式。
yourclass.s3_output.upload_fileobj(z, ...)
这同样适用于 zip 和 gz:
import boto3
import gzip
import pandas as pd
from io import BytesIO, TextIOWrapper
s3_client = boto3.client(
service_name = "s3",
endpoint_url = your_endpoint_url,
aws_access_key_id = your_access_key,
aws_secret_access_key = your_secret_key
# Your file name inside zip
your_filename = "test.csv"
s3_path = f"path/to/your/s3/compressed/file/test.zip"
bucket = "your_bucket"
df = your_df
gz_buffer = BytesIO()
with gzip.GzipFile(
filename = your_filename,
mode = 'w',
fileobj = gz_buffer ) as gz_file:
df.to_csv(TextIOWrapper(gz_file, 'utf8'), index=False)
s3.put_object(
Bucket=bucket, Key=s3_path, Body=gz_buffer.getvalue()
)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.