使用 Boto3/Python3 按列分隔，编写 s3 迭代到 csv 文件

Question

I am writing my output to a csv file. Here is my output:

bucket_name | s3_key_path

value        , value

bucket_name | s3_key_path

value        , value

I want the output to be:

bucket_name | s3_key_path

value        , value

value        , value

value        , value

Here is my code:

s3 = boto3.resource('s3',
    aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY
    )

bucket1=s3.Bucket(BUCKET)


def load_into_csv():

    with open('/my_path/ms3_files_V2.csv', 'w', newline='') as file:
        header=['bucket_name', 's3_key_path']
        for objects in bucket1.objects.filter(Prefix=PREFIX):
            bucket_name=  '{}' .format(bucket1.name)
            the_key= '{}' .format(objects.key)
            dd={'bucket_name': bucket_name, 's3_key_path': the_key}
            writer = csv.DictWriter(file, fieldnames=header)
            writer.writeheader()
            for row in dd:
                print(row)
                writer.writerow(row)
load_into_csv()

Any idea as to why my output keeps putting the headers for each value above it?

Answer 1

You need to put the writing of the header before the for loop:

        header = ['bucket_name', 's3_key_path']
        writer = csv.DictWriter(file, fieldnames=header)
        writer.writeheader()
        for objects in bucket1.objects.filter(Prefix=PREFIX):

Otherwise the header will be written for each object in the bucket.