Open a csv file from S3 in write mode and write content to the file

Question

I have function to get the billing amount by AWS account and by application using the Boto3 cost explorer. I get a json response as a result. Currently, I am converting the json to a csv file and storing it on my local machine. All works good.

However, I would like to have the csv file directly in S3. Like create a csv file in S3, create headers and copy the contents from json to csv. Is it possible to do this using s3.Object?

I am trying to use something like this:

bucket = s3.Bucket('my-bucket')

header = 'Account Number;Application;Amount\n'
with open('test.csv', 'wt') as file_object:
      file_object.write(header)

I dont get any errors, but no results either.

Updated question - This is how I am converting my json to csv to get only the columns that I need:

import boto3
import json
import csv
import yaml

client = boto3.client('ce',
      region_name='us-east-1',
      aws_access_key_id='AWS Access Key',
      aws_secret_access_key='AWS Secret Access Key')

response = client.get_cost_and_usage(
TimePeriod={
'Start': '2018-01-01',
'End': '2018-02-01'
},
Granularity='MONTHLY',
Metrics=[
'BlendedCost',
],
GroupBy=[
{
    'Type': 'DIMENSION',
    'Key': 'LINKED_ACCOUNT'
},
{
    'Type': 'TAG',
    'Key': 'Application'
},
],
)

response_ser = json.dumps(response)

# decode the unicode strings without converting the datatype
decode_response = yaml.safe_load(response_ser)

# get the first object in the response json
results_obj = decode_response['ResultsByTime']
results_dict = results_obj[0]

#get the groups
response_groups = results_dict['Groups']

raw_data = open(local_file, "w")
csvwriter = csv.writer(raw_data)
fieldnames = ['Account Number', 'Application', 'Amount']
writer = csv.DictWriter(raw_data, fieldnames=fieldnames)
writer.writeheader()

for i in response_groups:
accountNumber = i['Keys'][0]
application = i['Keys'][1]
amount = i['Metrics']['BlendedCost']['Amount']
writer.writerow({'Account Number': accountNumber, 'Application':
application, 'Amount': amount})

raw_data.close()

Hence, instead of creating the csv file on my local machine, I would like to have it created in S3. I am able to connect to the S3 bucket with the correct keys and can upload a file from my machine to S3. However, when I try to create a file in S3, it doesnt work. Am I doing something wrong here?

Answer 1

pip install smart_open

>>> # stream content *into* S3 (write mode):
>>> with smart_open.smart_open('s3://mybucket/mykey.txt', 'wb') as fout:
...     for line in ['first line', 'second line', 'third line']:
...          fout.write(line + '\n')

Answer 2

Not sure about your issue here but try Dictreader like this

with open(full_output_path, 'wb') as f:
        fields = ['email', 'lastname', 'firstname', 'groups']
        w = csv.DictWriter(f, fields)
        w.writeheader()
        w.writerow(row_data)

Answer 3

Hey so I have quite a lot of experience using boto3. You can definitely write csv directly to s3, however it uses boto3 s3 client not resource or object. From what I am getting about your question, you want to get the json output and convert it into a csv file storing it on s3 instead of your local computer. You can do this by using the data that you would normally create in the local file but it would be something like so:

client = boto3.client('s3')
variable = b'csv, output, from, json' # need bytes here or a file path
response = client.put_object(Bucket='bucket-name',Body=variable,Key='filenameyouwant.csv')

Also I'm not sure if you are appending result or overwriting, but you can dynamically add csv file data to the existing s3 object by using get_object . This will allow you to read the stream, append new data, and then you can re-upload all without writing to your local hard disk.

Please let me know if this did or did not answer your question

see documentation below: put_object