使用 spark 写入文件并使用 python 读取

Question

writing a file s3 using spark usually creates a directory with 11 files success and the other file name starts with name as part which has actual data in s3, how to load the same file using pandas dataframe since the file path changes because the file name Par for all 10 files with actual data varies in each run.

For example the file path at the time of writing:

df.colaesce.(10).write.path("s3://testfolder.csv")

The files stored in the directory are:

- sucess
- part-00-*.parquet

I have a python job which reads the file to pandas dataframe

pd.read(s3\\..........what is the path to specify here.................)

Answer 1

when writing files with spark, you cannot pass the name the file (you can, but you end up with what you described above). if you want a single file to later load to pandas, you would do something like this:

df.repartition(1).write.parquet(path="s3://testfolder/", mode='append')

The end result will be a single file in "s3://testfolder/" that starts with part-00-*.parquet . You can simply read that file in or rename the file to something specific before reading it in with pandas.

Answer 2

Option 1: (Recommended)

You can use awswrangler . Its a light weight tool to aid with the integration between Pandas/S3/Parquet. It lets you read in multiple files from the directory.

pip install awswrangler

import awswrangler as wr

df = wr.s3.read_parquet(path='s3://testfolder/')

Option 2:

############################## RETRIEVE KEYS FROM THE BUCKET ##################################

import boto3
import pandas as pd

s3 = boto3.client('s3')

s3_bucket_name = 'your bucket name'
prefix = 'path where the files are located'

response = s3.list_objects_v2(
    Bucket = s3_bucket_name, 
    Prefix = prefix 
)

keys = []
for obj in response['Contents']:
    keys.append(obj['Key'])
    
##################################### READ IN THE FILES  ####################################### 


df=[]
for key in keys:
    df.append(pd.read_parquet(path = 's3://' + s3_bucket_name + '/' + key, engine = 'pyarrow'))