在 Spark 中并行读取来自不同 aws S3 的多个文件

Question

I have a scenario where I would need to read many files (in csv or parquet) from s3 bucket located different locations and with different schema.

My purpose of this is to extract all metadata information from different s3 locations and keep it as a Dataframe and save it as csv file in s3 itself. The problem here is that I have lot of s3 locations to read the files(partitioned). My sample s3 location is like

s3://myRawbucket/source1/filename1/year/month/day/16/f1.parquet
s3://myRawbucket/source2/filename2/year/month/day/16/f2.parquet
s3://myRawbucket/source3/filename3/year/month/day/16/f3.parquet
s3://myRawbucket/source100/filename100/year/month/day/16/f100.parquet
s3://myRawbucket/source150/filename150/year/month/day/16/f150.parquet    and .......... so on

All I need to do is to use spark code to read these many files (around 200) and apply some transformations if required and extract header information, count information, s3 location information, datatype.

What is the efficient way to read all these files(differenct schema ) and process it using spark code (Dataframe) and save it as csv in s3 bucket? Please bear with me as I am new to spark world. I am using python (Pyspark)

Answer 1

I think what you want to do is use some Python/Pandas logic and parallelize the jobs with Spark. Fugue is a good fit for that. You can port you logic to Spark with very minimal code changes. Let's just worry about defining the logic with Python and Pandas first, and then we can bring it to Spark.

First the setup:

import pandas as pd

df = pd.DataFrame({"x": [1,2,3]})
df.to_parquet("/tmp/1.parquet")
df.to_parquet("/tmp/2.parquet")
df.to_parquet("/tmp/3.parquet")

We need a small DataFrame with all the files to orchestrate the jobs with Spark. For example:

file_paths = pd.DataFrame({"path": ["/tmp/1.parquet",
                                    "/tmp/2.parquet",
                                    "/tmp/3.parquet"]})

Now we can create a function that holds the logic for each file. Note that when we bring it to Spark, we will make 1 "job" per file path. Our function only needs to be able to handle one file at a time.

def process(df:pd.DataFrame) -> pd.DataFrame:
    path = df.iloc[0]['path']
    
    tmp = pd.read_parquet(path)
    
    # transformation
    tmp['y'] = tmp['x'] + 1
    
    # save
    tmp.to_parquet(path)
    
    # summary stats
    return pd.DataFrame({"path": [path],
                         'count': [tmp.shape[0]]})

We can test the code:

process(file_paths)

Which gives us:

path    count
/tmp/1.parquet  3

Now we can bring it to Spark using Fugue. We only need the transform() function to bring the logic to Spark. The schema is a requirement for Spark.

import fugue.api as fa
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

out = fa.transform(file_paths, process, schema="path:str,count:int", engine=spark)

# out is a Spark DataFrame
out.show()

The output will be:

+--------------+-----+
|          path|count|
+--------------+-----+
|/tmp/1.parquet|    3|
|/tmp/2.parquet|    3|
|/tmp/3.parquet|    3|
+--------------+-----+