使用 pyspark 从 S3 读取 parquet 分区表正在从分区列中删除前导零

Question

I have written a pyspark dataframe as parquet to s3 using EMR(pyspark), this data is partitioned by column(A) which is StringType()

in S3 the data looks something like this

table_path:
       A=0003
           part-file.parquet
       A=C456
           part-file.parquet

While I am reading this back as dataframe using pyspark I am loosing leading zeros in column 'A' of the datafram. Here is how the data is looking like

df =  spark.read.parquet(table_path)
df.show()

| A  | B |
| 3  | ..|
|C456| ..|

I don't want to loose the leading zeros here. The expected result is:

| A  | B |
|0003| ..|
|C456| ..|

Answer 1

Found the solution of this issue in the delta documentation.

Spark has a property enabled by default. With this property spark tries to infer the schema of the partition column. For a partition column of string type we can easily go ahead and switch it off.

# Update partition data type infer property
from pyspark.conf import SparkConf
from pyspark.sql import SparkSession, Window
conf = (SparkConf().set("spark.sql.sources.partitionColumnTypeInference.enabled", False))
sc=SparkSession.builder.config(conf=conf).getOrCreate()