使用 S3 中的嵌套分区文件创建数据框，并在架构中加载具有分区列名称的数据框

Question

Nested S3 parition file As below :

s3://alex/covid/exposure_date=**2021-02-01**/aec600b87b9a467d9395d2f5c0e2eeaa.parquet
s3://alex/covid/exposure_date=**2020-05-01**/dec600b87b9a467d9395d2f5c0e2eeaa.parquet
s3://alex/covid/exposure_date=**2021-06-01**/efe600b87b9a467d9395d2f5c0e2eeaa.parquet
s3://alex/covid/exposure_date=**2021-08-01**/acd600b87b9a467d9395d2f5c0e2eeaa.parquet

Tried different ways to read :

Way 1:

df = spark.read.parquet("s3:/alex/covid/**/*.parquet")

Way 2:

df=spark.read.option("recursiveFileLookup","true").parquet("s3:/alex/covid/covid_contact_mock_data.parquet/")

way 3:

df = spark.read.parquet("s3://alex/covid//**/*.parquet")

I am expecting the partition column also one of the dataframe columns

In local tried to do the but getting error

df = spark.read.parquet("C:/Users/mthma/covid_contact_mock_data.parquet/exposure_date=2020-12-03/*")

the error:

Exception in thread "globPath-ForkJoinPool-6-worker-115" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
        at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
        at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:793)
        at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1215)
        at org.apache.hadoop.fs.FileUtil.list(FileUtil.java:1420)
        at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:601)
        at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1972)

Answer 1

You are having trouble with wildcards.

At least in the local case, you can expand them to individual filenames with this:

folder = "C:/Users/mthma/covid_contact_mock_data.parquet/exposure_date=2020-12-03"
for file in glob.glob(f'{folder}/*'):
    df = spark.read.parquet(file)
    print(df.head())

https://docs.python.org/3/library/glob.html#glob.glob

S3 is different. You should use the boto3 library to query S3 for individual path names.