Using AWS Lambda and boto3 to append new lines to text file objects in S3

Question

I'm trying to use a python lambda function to append a text file with a new line on a object stored in S3. Since objects stored in S3 are immutable, you must first download the file into '/tmp/', then modify it, then upload the new version back to S3. My code appends the data, however it will not append it with a new line.

BUCKET_NAME = 'mybucket' 
KEY = 'test.txt'
s3 = boto3.resource('s3')

def lambda_handler(event, context):
    try:
        s3.Object(BUCKET_NAME, KEY).download_file('/tmp/test.txt')
    except botocore.exceptions.ClientError as e:
        if e.response['Error']['Code'] == "404":
            print("The object does not exist.")
        else:
            raise
    with open('/tmp/test.txt', 'a') as fd:
        fd.write("this is a new string\n")


    s3.meta.client.upload_file('/tmp/test.txt', BUCKET_NAME, KEY)

The file is always appended with the new string but never with a new line. Any ideas?

UPDATE: This problem does not occur on linux machines or on a Mac. Lambda functions run on linux containers, which means the file in /tmp/ is saved as a Unix-formatted text file. Some Windows applications will not show line breaks on Unix-formatted text files, which was the case here. I'm dumb.

Answer 1

You need to specify the local file path

import boto3
import botocore
from botocore.exceptions import ClientError

BUCKET_NAME = 'mybucket'
KEY = 'test.txt'
LOCAL_FILE = '/tmp/test.txt'
s3 = boto3.resource('s3')

def lambda_handler(event, context):
    try:
        obj=s3.Bucket(BUCKET_NAME).download_file(LOCAL_FILE, KEY)
    except ClientError as e:
        if e.response['Error']['Code'] == "404":
            print("The object does not exist.")
        else:
            raise
    with open('/tmp/test.txt', 'a') as fd:
        fd.write("this is a new string\n")


    s3.meta.client.upload_file(LOCAL_FILE, BUCKET_NAME, KEY)

Boto3 doc reference: http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Bucket.download_file

Answer 2

Nice Post! Just an adjustment.. You should change the order of LOCAL_FILE and KEY in the parameters of the download_file method. The correct syntax is:

obj=s3.Bucket(BUCKET_NAME).download_file(KEY,LOCAL_FILE)

Also it would be nice if we delete de local file in case of file not found in the bucket. because if we dont remove the local file (if exists obviously) we may be adding a new line to the already existed local file. With the help of this function:

def remove_local_file(filePath):
import os
     # As file at filePath is deleted now, so we should check if file exists or not not before deleting them
    if os.path.exists(filePath):
        os.remove(filePath)
    else:
        print("Can not delete the file as it doesn't exists")

the final code starting in the 'try' could be like this:

try:
    obj=s3.Bucket(BUCKET_NAME).download_file(KEY,LOCAL_FILE)
except ClientError as e:
    if e.response['Error']['Code'] == "404":
        print("The object does not exist.")
        remove_local_file(LOCAL_FILE)
    else:
        raise
with open(LOCAL_FILE, 'a') as fd:
    fd.write("this is a new string\n")

s3.meta.client.upload_file(LOCAL_FILE, BUCKET_NAME, KEY)

Answer 3

You don't need to download and upload a file in order to overwrite a file in S3; To overwrite an existing object you can just upload the file with the same name and it will be done automatically ( reference ). Look into the put_object function ( S3 doc ).

So your code will look like this:

BUCKET_NAME = 'mybucket' 
KEY = 'test.txt'
# Use .client() instead of .resource()
s3 = boto3.client('s3')

def lambda_handler(event, context):
    try:
        # (Optional) Read the object
        obj = s3.get_object(Bucket=BUCKET_NAME, Key=KEY)
        file_content = obj['Body'].read().decode('utf-8')
        
        # (Optional) Update the file content
        new_file_content = file_content + "this is a new string\n"
        
        # Write to the object
        s3.put_object(Bucket=BUCKET_NAME, Key=KEY, Body=str(new_file_content))

    except botocore.exceptions.ClientError as e:
        if e.response['Error']['Code'] == "404":
            print("The object does not exist.")
        else:
            raise