如何在不将S3中存储的文件保存到服务器的情况下进行操作？

Question

I have the following python script that downloads two files from an S3 compatible service. Then merges them and uploads the output to another bucket.

import time
import boto3
import pandas as pd

timestamp = int(time.time())

conn = boto3.client('s3')
conn.download_file('segment', 'segment.csv', 'segment.csv')
conn.download_file('payment', 'payments.csv', 'payments.csv')

paymentsfile = 'payments.csv'
segmentsfile = 'segment.csv'
outputfile = 'payments_merged_' + str(timestamp) + '.csv'

csv_payments = pd.read_csv(paymentsfile, dtype={'ID': float})
csv_segments = pd.read_csv(segmentsfile, dtype={'ID': float})
csv_payments = csv_payments.merge(csv_segments, on='ID')
open(outputfile, 'a').close()
csv_payments.to_csv(outputfile)

conn.upload_file(outputfile, backup, outputfile)

However if I execute the script it stores the files in the folder of my script. For security reasons I would like to prevent this to happen. I could delete the files after the script was executed but let's assume my script is located in the folder /app/script/ . This means for a short time, while the script is being executed, someone could open the url example.com/app/script/payments.csv and download the file. What is a good solution for that?

Answer 1

The simplest way would be to modify the configuration of your web server to not serve the directory that you are writing to or write to a directory that isn't served. For example, a common practice is to use /scr for this type of thing. You would need to modify permissions for the user your web server runs under to ensure it has access to /scr.

To restrict web server access to the directory you write to you can use the following in Nginx -

https://serverfault.com/questions/137907/how-to-restrict-access-to-directory-and-subdirs

For Apache you can use this example -

https://serverfault.com/questions/174708/apache2-how-do-i-restrict-access-to-a-directory-but-allow-access-to-one-file-w

Answer 2

In fact, pandas.read_csv let you read a buffer or byte object. You can do everything in the memory. Either put this script in a instance, even better, you can run it as AWS lambda process if the file is small.

import time
import boto3
import pandas as pd

paymentsfile = 'payments.csv'
segmentsfile = 'segment.csv'
outputfile = 'payments_merged_' + str(timestamp) + '.csv'

s3 = boto3.client('s3')
payment_obj = s3.get_object(Bucket='payment', Key=paymentsfile )
segment_obj = s3.get_object(Bucket='segment', Key=segmentsfile )

csv_payments = pd.read_csv(payment_obj['Body'], dtype={'ID': float})
csv_segments = pd.read_csv(segments_obj['Body'], dtype={'ID': float})
csv_merge = csv_payments.merge(csv_segments, on='ID')
csv_merge.to_csv(buffer)
buffer.seek(0)

s3.upload_fileobj(buffer, 'bucket_name', outputfile )