EmptyDataError：从 S3 存储桶读取多个 csv 文件到 Pandas Dataframe 时，没有要从文件解析的列

Question

I have a source s3 bucket which has around 500 csv files, I want to move those files to another s3 bucket and before moving I want to clean up the data so I am trying to read it to a pandas dataframe. My code works fine and returns dataframes for a few files and then it suddenly breaks and gives me the error " EmptyDataError: No columns to parse from file " .

sts_client = boto3.client('sts', region_name='us-east-1')
client = boto3.client('s3')

bucket = 'source bucket'
folder_path = 'mypath'

def get_keys(bucket,folder_path):
    keys = []
    resp = client.list_objects(Bucket=bucket, Prefix=folder_path)
    for obj in resp['Contents']:
        keys.append(obj['Key'])
    return keys

files = get_keys(bucket,folder_path)
print(files)

for file in files:
    f = BytesIO()
    client.download_fileobj(bucket, file, f)
    f.seek(0)
    obj = f.getvalue()
    my_df = pd.read_csv(f ,header=None, escapechar='\\', encoding='utf-8', engine='python')
    # files dont have column names, providing column names
    my_df.columns = ['col1', 'col2','col3','col4','col5']
    print(my_df.head())

Thanks in advance!

Answer 1

Your file size is zero. Instead of os.path.getsize(file) use paginator to check as follows:

import boto3

client = boto3.client('s3', region_name='us-west-2')
paginator = client.get_paginator('list_objects')
page_iterator = paginator.paginate(Bucket='my-bucket')
filtered_iterator = page_iterator.search("Contents[?Size > `0`][]")
for key_data in filtered_iterator:
    print(key_data)