Trying to create CSV file directly in S3 and write data in CSV

Question

Hi all i am trying to create csv in S3 and feed data in the csv but an not able to archive it.

Below is the code

import boto3
import csv
import time

date=time.strftime("%d-%m-%Y")

BUCKET_NAME = 'bucketname'
PREFIX = '/'
filename = f'result{date}.csv'

sts_s3 = boto3.resource('s3', region_name='us-west-2', endpoint_url='https://sts.us-west-2.amazonaws.com')
obj = sts_s3.Object(BUCKET_NAME, filename)


with open(f"s3://{BUCKET_NAME}{PREFIX}{filename}", 'w', newline='') as f:
    column = ['1111','2222', '3333']
    thewriter = csv.DictWriter(f, fieldnames=column)
    writer = csv.writer(f)
    thewriter.writeheader()
    thewriter.writerow({'1111': 'aaa','2222':'bbb', '3333':'ccc'})

Answer 1

Here an example using s3fs and pandas. With pandas I create,handle the data and with s3fs upload to s3.----> pip install pandas and pip install s3fs

import s3fs
import pandas as pd

ACCESS_KEY='XXXXXXXXXXXXXXXXXXX'
SECRET_KEY='YYYYYYYYYYYYYYYYYYYYYY'

data = [['tom', 10], ['nick', 15], ['juli', 14]]
df = pd.DataFrame(data, columns = ['Name', 'Age'])

bytes_to_write = df.to_csv(None).encode()
fs = s3fs.S3FileSystem(key=ACCESS_KEY, secret=SECRET_KEY)
with fs.open('s3://mybuckets//file.csv', 'wb') as f:
    f.write(bytes_to_write)