从 Amazon S3 存储桶中的某个位置读取 AWS Sagemaker 中的多个 csv 文件

Question

I have multiple csv files in a location in S3. The name of those files is in a date format. Example: 2021_09_30_Output.csv

I need to understand how I can read all the files in this folder while selecting only the dates that I require. An example would be reading only the files from September. ie: "2022_09_*.csv" which would read only the files from that month

Would appreciate the help. Thanks

Answer 1

You can create a function that will return all files from a particular date onwards using the datetime library based on the naming convention of your files. The following snippet can get you started:

import datetime

s3 = boto3.resource('s3')
BUCKET_NAME = 'name'
september_1 = datetime.datetime(2021, 9, 1)
files = get_files_after(BUCKET_NAME, september_1)
for file in files:
    contents = file['Body'].read()
    contents = contents.decode("utf-8")
    ...


def get_files_after(bucket, date):
    files = []
    for obj in s3.Bucket(bucket).objects.all():
        key = obj.key
        file_date = key[:-4] # Remove '.csv' from name
        file_date = datetime.datetime.strptime(file_date, '%Y_%m_%d')
        if file_date > date:
            files.append(obj)
    return files